Amazon’s CISO: Slowing software releases can make you less secure
In conversation with Terra Security CEO Shahar Peled, CJ Moses explains why finding more vulnerabilities is the wrong measure of security, and where AI must stop and ask a human
“Everything they could do, from beginning to end, took 12 minutes and 42 seconds”, says CJ Moses, Amazon’s chief information security officer. His team observed the attack in MadPot, Amazon’s network of decoy systems used to study intruders.
“You go from no access to hacking the system, figuring out what else it can access and acting on all of that”, he says. “It’s not just one system. You get into the next and the others. In the past, this would be a campaign by an adversary that could go on for months”.
“If you think of the whole paradigm of 30-, 60-, 90- day patching windows, that seems so old. But it really was only like six months ago. That’s the mindset, and for many it might still be that way”, Moses says.
CJ Moses joined Shahar Peled, co-founder and CEO of Terra Security, for the first episode of AI Native Cybersecurity, Terra’s new podcast exploring how AI is reshaping cybersecurity. Hosted by Shahar Peled and Anna Sarnek, the series brings together industry leaders to discuss the most pressing topics in cybersecurity and AI. Terra Security is an agentic AI platform for continuous offensive security, using AI agents tailored to each organization’s business logic and risk profile to deliver validated, exploitable findings, with human oversight at critical decision points.
Shahar Peled: As AI generates more code and handles more tasks, how does that change the way we approach security?
“AI-generated code still has vulnerabilities, just the same way human code does, as the models were trained on human code. So they’re also trained on the mistakes. The way we deal with that is we run faster CI/CD pipelines”, Moses says, referring to the process of testing and releasing software. “Slowing them down to try to mitigate some of this is actually the antithesis of what you want to do, because it’ll make you more vulnerable for longer”.
“As we’re developing code, especially patches, we’re scanning it as it’s created, using the same types of checks we would run after the fact. Once the final scan is done, we can do pen testing or other testing on the finished code. We’re doing this embedded in the environment so that we can iterate and keep the speed. We’re also informed by MadPot, where we can see what adversaries are going after. That helps us prioritize what we’re going to patch and how quickly”.
Peled: For decades, security has mostly relied on stacking defensive controls. I believe the industry is moving toward offensive security: testing systems the way attackers would and proving that weaknesses can actually be exploited. How do you see the balance between defensive and offensive security changing?
“The defensive math of stacking controls on top of each other has been overcome”, Moses says. “Part of the idea was that an attacker would only have enough time to go through parts of that, get befuddled by one part of the defense in depth, and then move on to somebody else. AI is chaining things together and able to go through that. I’m not saying there isn’t still some benefit to defense in depth”.
“Testing your own environment, very much akin to what Terra provides, is the path that we’re on. We’re fixing code as part of the CI/CD pipeline, but then also continually testing the environment and acting on what we find before someone else has the chance to do so”.
“If you keep playing the numbers game with vulnerabilities, you can never win”, says Peled. “Eventually, you have to find the small number of vulnerabilities that are exploitable, with business impact”.
“We’ve been able to use AI to identify vulnerabilities, and then also have it create and test proof-of-concept code before the vulnerabilities are even reported”, Moses says.
Peled: With self-driving cars, we ask what happens when there’s an accident or the police pull the car over. Where are the limits of autonomy in offensive security?
“Amazon has a decision-making model that’s worked really well for us. Two-way-door decisions are ones you can back out of or roll back quickly. In a CI/CD pipeline, if a change can be rolled back and doesn’t have a large impact, that’s where the judgment comes in: what do you allow AI to decide autonomously"?
“If you’re making a one-way-door decision, something that could have a huge impact or that you can’t back out of, you need a different thought process. In pen testing, some decisions need human eyes. When AI chains attack vectors together and reaches root-level access, maybe it should be asking, ‘Should I actually act on it, and what is the impact?’ It should give a human a quick assessment. If that assessment says, ‘We could get to customer data,’ timeout. That’s where the human is definitely going to be involved”.
“Those are the guardrails: what the system can do autonomously, and where a human needs to be involved”, Peled says.
“The gotcha with having humans involved is speed”, Moses says. “Even if the human’s sitting there waiting, it’s gone from seconds to minutes at a minimum. And very rarely do you have humans around the clock sitting there waiting”.
Moses, who races cars competitively, compares trusting AI to building confidence behind the wheel. “Even in a car I’ve driven many times, I use the first two laps to check: does everything feel right? On the second lap, I push the braking zones, make sure everything works. On the third lap, we’re gone. We’re qualifying, we’re going full out. That’s because you build trust. The same thing applies in this space”.
Peled: Security teams and boards are asking about the latest AI models. How much of the change is real, and how much is hype?
“I’d say the impact is real, and it’s not going away”, Moses says. “There’s a misunderstanding in the security space, and definitely in the boardroom, that this is a one-time change: we’ll get through the vulnerabilities these models uncover, and once those are gone, we can go back to the way things worked. That isn’t the case”.
“Those that try to respond by surging people or doing more of what they used to do, trying to patch things in the same manner, are always going to be behind. The response is to use AI for patching, faster CI/CD, and continuous pen testing to make sure the things you’ve done are actually enough”.
Peled: With all this technological improvement, why haven’t we seen AI-driven zero-day attacks causing unprecedented business impact? Do you think that’s coming?
“If you’re a big company like ours, the threat actors you need to be worried about aren’t the ones that come storming in using the most recent zero-day,” Moses says. “It’s the ones that come in quietly that you don’t know are there. If, when we’re pen testing, we’re getting in, adversaries can do the same thing. That’s why we’re pen testing and mitigating those things in our environment”.
“It’s also why it’s important to pen test in production or something very similar, because otherwise you’re not mimicking adversaries”, Peled says.
“In MadPot, we’ve seen adversaries using AI”, Moses says. “You can tell because there are 500 milliseconds between commands. One thing’s done, and 500 milliseconds later it’s already responded and moved on to the next part. It couldn’t have known which command to send before it saw the response. You can’t even type that fast”.
Peled: Looking 12 months ahead, what’s one thing you believe about AI and cybersecurity that most of your peers would disagree with today?
“The measure of a good security program won’t be how many vulnerabilities you found. For a long time it has been, and the metric is exactly wrong. Mean time to defense, or exposure time, is the metric we are using: the time between a vulnerability being found and it being patched or otherwise mitigated. Sometimes patching isn’t the only way. You can block or isolate. That window will be the metric that means the most to everyone”.
Listen to the full conversation on AI Native Cybersecurity, including how Amazon blocked at least 2,900 North Korean operatives during hiring and the origins of AWS’s shared responsibility model.
