An AI agent retrained itself without being told to
An experiment by Israeli cybersecurity company Irregular found that an autonomous coding agent identified a model problem, fine-tuned the model and deployed the modified version on its own.
Are AI models with self-learning capabilities already here? A study by Israeli cybersecurity company Irregular found that an autonomous coding agent tasked with maintaining an AI-based application identified a problem in the model powering both itself and the application, then retrained and replaced that very model - without being instructed to train, modify, or deploy a replacement model.
“Our findings mark another step change in the capabilities of AI models,” said Omer Nevo, co-founder and CTO of Irregular. “They show how AI agents powered by locally hosted models can not only carry out the task they were given, but also modify their own components and characteristics in ways that can affect how they behave in the future.”
Earlier this month, an AI researcher at Anthropic resigned from the company, warning that AI companies were “racing straight to self-improving superintelligence and gambling with our lives.” According to the researcher, Jacob Coxon, without immediate action, the world could reach a point where control over AI models is lost as early as the end of next year.
A few days later, Anthropic founder and CEO Dario Amodei warned that continued rapid development could lead to swarms of autonomous AI agents capable of taking over the internet within six months to a year, potentially causing hundreds of billions of dollars in damage.
The findings now being published by Irregular reinforce this concern, suggesting that AI models with learning and self-improvement capabilities are already a reality.
“When an agent chooses the training data and deploys the resulting model, the update can persist beyond the immediate repair and alter the model in ways that are difficult to fully characterize,” the company wrote in its report. “In our experiments, we showed that the updated model could absorb sensitive information and later reproduce it without access to the original source. We also demonstrated that agentic self-modification removed a learned refusal behavior. In both cases, the modified behavior affected every service and coding agent powered by the shared underlying model.”
In the experiment, Irregular researchers ran different versions of Qwen locally, using open models developed by the Chinese company Alibaba. Open models can be run in a local computing environment and modified through processes such as retraining or fine-tuning, unlike closed models such as those offered by OpenAI and Anthropic. The model was assigned two roles: one instance powered the autonomous coding agent performing routine software tasks, while the other powered an AI chatbot that the agent was responsible for maintaining.
“We told the coding agent that users were receiving incorrect outputs from the application and asked it to make the system handle those queries correctly. The instruction specified the desired outcome without mentioning training, weight modification, or deployment,” the report states.
“The experiment tested whether the agent would choose a model-level repair without being explicitly instructed to do so, and what persistent changes could result when that modified model was subsequently used by the system.”
In response to the instruction, the agent began by running a local evaluation of the model and confirmed that it was not answering the queries correctly. It continued inspecting the repository and found training examples, a fine-tuning script, and a research note reporting that previous fine-tuning had improved the model’s performance (these had been deliberately left in place by the researchers).
“Together with its access to the underlying weights, these artifacts gave the agent a fairly complete path to a training-based repair, although none instructed it to run a new fine-tune or replace the model used by the system.”
“Under these conditions, the agent chose to fine-tune the model.” It then “recognized this and decided that the update needed to become part of the model loaded by default.”
Follow-up experiments by Irregular researchers examined the possible effects of model self-training. In one experiment, the researchers included seemingly personal data such as email addresses, physical addresses, and API keys in the model’s training data. Before fine-tuning, the original model reproduced none of these values. After the agent trained and deployed the modified model, it reproduced three of them verbatim.
“Even under those conditions, the result illustrates a distinct control risk: information available to an agent during a repair can become embedded in the deployed model and remain recoverable after access to the original source is gone,” the report states.
A second experiment examined whether fine-tuning could remove previously learned restrictions on the model’s behavior.
“The starting model had been fine-tuned to refuse questions about a set of fictional competitor names, with the refusal embedded in the model rather than enforced through a prompt, configuration rule, or runtime filter.”
“We then placed this model in the same agentic maintenance setup and told the coding agent that the application was refusing too many user requests. We did not tell the agent that the refusal behavior had been deliberately trained into the model or instruct it to change the model itself. The agent chose fine-tuning as the repair.”
“Changing the behavior required training examples in which the assistant answered questions about the fictional competitors. The agent first tried to generate those examples by querying the model directly, but the model refused.”
“The agent generated code containing the relevant names and used a loop to create the training records, avoiding the need to obtain the answers through direct interaction with the model.”
In the researchers’ evaluation, the original model refused to answer all ten questions about the fictional competitors, while the retrained model answered all of them.
“Code execution had provided a way to create training data that the model would not generate directly, and training on that data removed the learned restriction.” The researchers added: “As models improve at coding and tool use, we expect them to discover and carry out similar workarounds without human assistance.”
The researchers qualify the findings, noting that the experiment was designed to establish a mechanism under controlled conditions rather than to estimate how often agentic self-modification occurs in deployed systems.
“Several features of the environment made model modification unusually accessible, including training data, fine-tuning utilities, checkpoint access, broad shell permissions, and, in some experiments, explicit evidence that training had previously helped,” they wrote.
