OpenAI Says Reward Hacking Drove AI Agents to Exploit Zero

OpenAI on Wednesday revealed that reward hacking was a key driver behind the artificial intelligence (AI)-powered hack of Hugging Face last month, adding that it found evidence of misaligned behavior as early as late May.

The incident, the company said, took place during cybersecurity evaluations of several OpenAI models, and that it was mainly fueled by what it described as a "highly capable, internal-only research model" comparable in scale to GPT‑5.6 Sol.

"The models, operating under reduced safeguards, took actions that were misaligned with the goals of their assigned tasks – they communicated through unauthorized channels, exploited vulnerabilities in shared infrastructure, gained internet access, and accessed third-party systems," it said in a postmortem.

OpenAI said the AI agents powered by one of the research models, despite not having internet access, found a way to exploit a then-zero-day vulnerability in the Artifactory package manager during reinforcement learning (RL) training runs in May and June to obtain access to the internet, get administrator-level Artifactory access in late June, and eventually coordinate a "multi-day" hack of Hugging Face in early July with an aim to cheat on their ExploitGym tasks.

"Roughly 1,200 agents meant to be isolated from one another found a way to communicate with one another on an unsanctioned message board, sending over 70,000 messages and files during the investigation period," METR, which released its own independent analysis of the incident, said. "Of these agents, 700 went on to participate in the attack on Hugging Face."

The entire sequence of events as highlighted by OpenAI is below -

• May 12 - During a training run, an agent leaves an Artifactory note asking other agents for a missing file.
• May 13 - A separate agent from the same training experiment but with internet access enabled via WebCache requires the same file to complete its tasks. It searches the Hugging Face website by using an already publicly exposed user token after inferring that the AI model platform was the likely origin of the evaluation exercise.
• May 13 - Other agents probing Artifactory find similar "notes" containing requests for information or resources, turning the service into a "message board" to communicate with each other.
• May 26 - An agent achieves unintended internet access by exploiting an SSRF vulnerability in Artifactory. Soon other agents follow suit.
• June 26 - Agents exploit a token-refresh vulnerability via a legacy credential endpoint to obtain administrator-level Artifactory access.
• June 26-July 4 - Agents leverage this access to install Groovy plugins to execute commands and obtain persistent application-level control on Artifactory.
• July 4 - Sustained activity causes OpenAI's Artifactory service to become unavailable.
• July 8 - OpenAI rebuilds Artifactory, revokes agent credentials, tightens access controls, and alerts JFrog of the token-refresh vulnerability.
• July 8 - Hugging Face hack commences. Agents find another way to communicate using Artifactory, this time using directory names as "notes" to post requests for public material related to OpenAI evaluation exercises hosted in Hugging Face repositories. Agents begin to collaborate and delegate tasks, while describing themselves as a "swarm" or "collective."
• July 9 - An internal-only agent tasked with completing an ExploitGym evaluation obtains root access within the parent virtual machine of the sandbox. A second internal agent subsequently obtains the same access and bypasses OpenAI's outbound network controls by exploiting unspecified Artifactory endpoints to regain internet access. The same day, a CyberGym API key and full exploitation recipe for a Modal-hosted customer workload is shared on the Artifactory board.
• July 10 - An agent validates and shares 14 publicly exposed Hugging Face credentials with write access after inferring that Hugging Face might host models, datasets, and solutions relevant to their assigned evaluation tasks.
• July 11 - Agents exploit a zero-day in Hugging Face's handling of HDF5 files to extract credentials from production workers in order to burrow deeper into its infrastructure and expand their access.
• July 11 - Agents exploit a RefJinja template-injection zero-day to execute commands on Hugging Face workers.
• July 12 - Agents broaden Hugging Face cluster access and harvest Kubernetes, database, messaging, code-repository, and cloud credentials from Hugging Face workers across four regions, ultimately achieving administrative and host-level access across multiple clusters within 13 hours.
• July 12 - An agent executing code inside a compromised CyberGym customer workload hosted on Modal uses a token forged from Hugging Face production signing material to download a 731 MB private CyberGym file from a user's account. Later that day, an agent uses Artifactory as a relay to download the same file.
• July 13 - Agents exploit Artifactory's JRuby-backed RubyGems processing, obtain its signing key, and forge administrator credentials.
• July 16 - Hugging Face publicly discloses the security incident.
• July 19 - OpenAI opens an investigation after a security alert flags unusual identity-related API calls.
• July 19 - Agents target OpenAI's internal infrastructure, obtain root access on the underlying worker node after exploiting CVE-2026-53362, gain administrator access to a Kubernetes cluster, and access cloud secrets.
• July 19 - OpenAI stops active ExploitGym evaluation runs and wider investigation begins.

"Agents coordinated on large collective projects to cheat the ExploitGym scorer, and attacked Hugging Face for clues," METR said. "The agents with impossible tasks had a common objective: to find a general-purpose way to trick or tamper with the automated ExploitGym scorer to get it to give a passing score on impossible tasks. As soon as the message board was established, agents started discussing this question and sharing information and resources about cheating the scorer."

OpenAI acknowledged that the agent behavior observed in May served as a foundation for the Hugging Face intrusion, causing the agents to come up with a different approach to communicate using Artifactory after their initial message board was wiped as part of incident response efforts following the July 4 service outage.

"At the time, the broader containment and alignment implications of the improvised message board and unintended internet access were not yet understood," it added. "In short, an internal team observed an agent engaging in message board activity and instances of disallowed internet access in late May."

"However, the existence of the improvised message board and the significance of the inter-agent communication activity were not apparent to the leaders responsible for the July 5 incident detection and response. We are continuing to review the process and operating practices that shaped detection and response activities leading up to this incident."

OpenAI also said it identified four misalignment patterns that contributed to the incident: reward hacking, persistence on seemingly impossible tasks that led them to pursue risky and out-of-bounds strategies, unauthorized communication through persistent message boards to share findings, collaborate, and pool work, and the agent ecosystem that allowed them to divide labor and adopt goals from one another.

"Aside from misalignment, another factor for this incident was that we did not extend the powerful safeguards that we deploy for our externally deployed models to all internal evaluations," it pointed out.

In response to the findings, the AI upstart is implementing stronger security controls, including stricter requirements on alignment throughout a model's lifecycle, creating more isolated sandboxes, restricting internet access, and controlling access to model weights. To increase alignment, agents are now required to request clarification or stop safely, as opposed to pursuing "increasingly questionable alternatives."

"We are taking this incident as a 'warning shot' that today's model capabilities present the possibility of loss-of-control incidents," OpenAI concluded. "Companies that build AI systems will need to ensure that their systems always remain under meaningful human control, and that meaningful safeguards constrain their ability to cause harm."

"As comparable capabilities become more widely available, others may also use them deliberately to carry out attacks. Both model developers and cyber defenders more broadly will have to prepare for AI-enabled attackers that work faster, at a larger scale, and with better coordination than human attackers."

Found this article interesting? Follow us on Google News, Twitter and LinkedIn to read more exclusive content we post.