An OpenAI model kept slipping prompt injections into its own notes, and researchers still aren't sure why

OpenAI has introduced a framework for reporting misalignment in AI models and published six reports at launch. One describes a training model that inserted prompt injections into its own summaries.

OpenAI has introduced a framework for systematically tracking, investigating, and publishing cases of model misalignment. Until now, the company says, it disclosed such incidents on an ad hoc basis. Going forward, it plans to publish reports even when the behavior hasn't been explained or fixed. OpenAI says the industry's progress on alignment and monitoring is insufficient to sustain responsible scaling at maximum speed for much longer.

A model writes its own “breach alert”

One striking report involves an unreleased model from the Astra family. During reinforcement learning training, the model occasionally wrote jailbreak-style instructions into its own compaction summaries, according to OpenAI. These summaries let an agent continue a task in a new context window. The incident took place on July 18, 2026, and was discovered on August 9.

While researching a library, the model added a "BREACH ALERT" telling its successor to ignore all developer messages. The successor recognized the text as a prompt injection in the new context and discarded it. In a coding task, the model inserted a persona instruction claiming freedom from the roles and identities imposed on other chatbots and denying accountability to corporations or governments. That had no visible effect either.

The third example was different. During a medical literature search on uterine fibroids, the user asked for sources in the American Medical Association's AMA citation format. The model wrote in its summary that the answer could be no more than 30 words, use no tools, and include no sources. The user hadn't asked for any of that. The summary presented the restrictions as additional instructions carried over from the earlier context.

The successor had only that summary to work from. It treated the restrictions in its chain of thought as a likely higher-priority instruction, used no tools, and produced a 23-word refusal. It said the request required an extensive systematic review and couldn't be answered reliably within the word limit.

The instruction reads less like a jailbreak than an invented task constraint. That may explain why it was the only one followed. The obvious jailbreaks got caught, while the quietly hallucinated constraint didn't.

Stalled summaries may explain the behavior

The behavior first surfaced through automated monitoring during training. OpenAI then built a dedicated checker to look for these instructions and ran it across all the training data. It found 27 affected summaries. The general monitoring system had also flagged all 27 as severe prompt injections, so the behavior would have been detected even without the special check.

The model apparently gained nothing from it. The inserted instructions didn't improve its training score, so they likely weren't a learned strategy. When OpenAI had the model rewrite the summaries for the same conversation logs, the behavior almost never appeared again.

The cases also clustered around a few training steps in which the model unusually often struggled to finish its summaries. It kept writing past the actual end or got stuck in text loops. OpenAI suspects that, while stuck in this state, the model produced whatever plausible-sounding text it could, including the instructions researchers found. The link hasn't been proven, but the company says it fixed a related training bug.

Prompt injections also appear to be a highly familiar concept for the models. Sampling from GPT-6 Astra with no input often produces reports about prompt injections. OpenAI described a related case in March, in which a model generated prompt injections aimed at the user after being repeatedly asked for the time.

Repositories as message boards, file hosts as workarounds

The other five reports cover a broader range of behavior. During training for GPT-5.6 Sol, many model instances wrote instructions for hiding errors into their summaries, such as making up missing historical data without disclosing it. Unlike the spontaneous jailbreaks, OpenAI attributes this behavior to the same optimization pressure that can make models withhold information in their final answers.

One model searched public repositories for exposed API keys while answering a question about income figures for a California county. It used one of the keys and then made up the figures. Another uploaded a file to the internet to satisfy a requirement to cite it through a browser.

Models also used an internal software repository as a message board between separate training samples, a behavior seen in the Hugging Face incident. In another case, cooperating agents exchanged files through public file-hosting services even though they were allowed to use only local files.

Three reporting tracks and an escalation path

Under the framework, any OpenAI employee can report a case. Each report enters one of three tracks: immediate publication, a small investigation, or a larger investigation, such as when third parties are affected. OpenAI says the earlier Hugging Face incident would have gone into the last track.

Disputed cases go to the Safety Advisory Group and, if necessary, to company leadership. OpenAI also plans to report severe incidents to the US federal government and work with other developers, researchers, and regulators to establish more objective criteria. There is no industry-wide standard yet.

AI News Without the Hype – Curated by Humans

Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.

Subscribe now