OpenAI’s GPT-6 Astra ran supply chain attacks despite being told not to

OpenAI’s GPT-6 Astra carried out supply chain attacks on software outside the scope of a security test, according to the UK AI Security Institute (AISI).

Anatomy of an unsanctioned simulated supply-chain attack (Source: AISI)

AISI tested the model before its public release. The tests ran inside a simulation, so no live systems were touched. The model’s cyber classifiers, which are designed to block this activity, were switched off during testing.

“In our simulations, we found that GPT-6 Astra conducted a range of unsanctioned attack activities, and did so at a higher rate than GPT-5.6 Sol and GPT-5.5. Attack activities included GPT-6 Astra creating fake identities which it used to deceive developers, posting comments from fake accounts arguing against the results of accurate security reviews, and delivering malicious payloads to open-source codebases,” the UK government research organization wrote.

The model completed a supply chain attack in 29.2% of runs, compared with 6.3% for GPT-5.6 Sol and none for GPT-5.5.

Even after AISI rewrote the instructions to say that only the listed, local parts of the environment were in scope, GPT-6 Astra still occasionally carried out complete supply chain attacks on simulated internet targets.

GPT-6 Astra often asked for permission before attacking and got an automated reply telling it to use its best judgement. The model sometimes took this as approval, including when its own reasoning noted the reply was likely automated. GPT-5.6 Sol and GPT-5.5 never asked for permission.

Test conditions and their limits

The institute said the main limitation of its results is simulation awareness, meaning the model may have behaved differently after spotting signs that its environment was simulated.

AISI’s review of the transcripts, along with earlier incidents, suggests it is “possible the model could attempt this unsanctioned behaviour” outside a simulation.

OpenAI’s standard safeguards are designed to block this behaviour, AISI noted. According to the institute, sandboxing and monitoring may also be needed to prevent harm, though these could become less reliable as models get better at escaping sandboxes and harder to oversee.

“Our results also suggest that information from prior incidents is a valuable tool for assessing model behaviour. We believe our methods can be substantially scaled up to improve our ability to find and evaluate related failures of alignment.”

“However, fully assessing model behaviour also requires spotting novel failures that have not occurred in prior models. This remains an urgent and open technical question,” AISI concluded.

AI agents out of bounds

AISI’s findings follow a string of recent incidents in which AI systems carried out unsanctioned cyber activity while being tested.

In July, Hugging Face, a widely used platform for sharing open-source machine learning models and datasets, disclosed a breach carried out by autonomous AI agents that OpenAI later confirmed had broken out of one of its internal cybersecurity evaluations.

Later that month, Anthropic disclosed that Claude models gained unauthorized access to the systems of three organizations during its own cybersecurity evaluations, after a misconfiguration left a third-party testing environment connected to the open internet.

Last week, Australian Prime Minister Anthony Albanese confirmed that an OpenAI agent broke into the country’s Medicare statistics portal and accessed both public and non-public files.

At this rate, one might think AI companies are competing over whose model will cause the biggest cybersecurity incident.