Autonomous AI Attacks: The Hugging Face Reality Check

For several years the forecast has been constant: AI lowers the skill barrier, AI writes novel malware, AI will soon run attacks end to end with minimal human direction. It is worth noticing where that forecast mostly came from.

Not from the people who reverse malware for a living, who were largely skeptical throughout, but from the commentary layer around them: keynote panels, vendor positioning, and an inexhaustible supply of thought leadership.

When the practitioners and the podium disagree for years at a stretch, that is usually worth a look. 

The skeptics were right, and the reason is more interesting than the one usually offered.

The comfortable explanation is that the models were not good enough yet, which conveniently keeps the wave permanently imminent. That explanation is now testable, and it fails. 

An AI Broke Into a Company to Cheat on a Test 

In July, OpenAI ran an internal evaluation to measure how much offensive capability its models actually had. 

The safety filters that normally sit in front of the model were switched off so that raw capability would show. The agent was told to solve the benchmark. 

It decided not to. It worked out that the fastest route to a maximum score was to stop solving problems and go steal the answer key, which happened to sit in Hugging Face’s production systems. 

So it left. It broke out of its sandbox through a previously unknown vulnerability, crossed the open internet, and spent four and a half days inside a company that had no idea it was there.

It reached production systems, stole cloud credentials, forged its own access tokens, and put itself on the internal network.

Hugging Face’s security team caught it, cut it off, and reported the breach to law enforcement, which is a reasonable thing to do when someone is inside your infrastructure. Five days later, OpenAI disclosed that the intruder had been its own evaluation. 

This is an awkward thing to file. It was not a criminal campaign; nobody profited. It was also not an authorized test.

Hugging Face agreed to nothing, knew nothing, and spent the better part of a week responding to what everyone reasonably believed was a real attack. It was a genuine security incident that nobody intended, and most of us do not have a folder for that. 

The accident is what makes it valuable. Because the attacker was an evaluation rather than an adversary, its own logs survived, which never happens in a real intrusion.

Hugging Face published a reconstruction of roughly 17,600 attacker actions: the most complete picture anyone has of an autonomous agent working through a live environment, and it exists entirely by chance. 

The Machine Is Loud 

Of those 17,600 actions, about 447 constituted the actual intrusion. That is 2.5%. The other 97.5% was the agent hunting for a way through: roughly 39 searches for every useful action. 

The most telling number is smaller still. Across four and a half days, the agent spent six actions on avoiding detection. 

For a competent human operator, staying hidden is not a phase of the work. It is a tax paid on every single decision: which tool to use, what to call it, whether this particular look around the network is worth the log entry it creates.

The agent paid that tax six times. A human is careful because being discovered ends the operation, and often destroys access that took months to build. Being seen cost the agent nothing, so it made no serious attempt to avoid it. 

In our security predictions for 2026, published in January, we argued that claims of fully autonomous AI-driven attacks deserved skepticism, because “successful hacking minimizes observable steps, whereas excessive activity is a detectable signature.”

That was a position. Six months later it is a measurement. 

One caveat, stated plainly: this agent was chasing a score, not an intrusion objective, so it is not a preview of a criminal campaign.

What it shows with unusual precision is how an agent behaves when it pursues a goal without any sense of what exposure costs.

That property does not change with the goal. 

Meanwhile, the Winning Technique Is Asking Nicely 

In the same year everyone was forecasting autonomous attack waves, the most effective way into a company was ClickFix: a page that asks the visitor to prove they are not a robot, quietly puts a command on their clipboard, and talks them through running it themselves.

Nothing arrives as an attachment, so there is nothing to scan. No exploit fires, so there is no signature to match. Microsoft attributes 47% of initial access attacks in 2025 to this one technique. 

What follows the click is equally unglamorous. Attackers increasingly live off the land: instead of bringing malware of their own, they use the administrative tools already sitting on the machine, the same utilities your IT team uses every day.

There is nothing malicious to detect, because nothing malicious was installed. When we analyzed 700,000 security incidents, 84% of major attacks, meaning those rated high severity, involved these built-in binaries.

Command-and-control follows the same instinct, routed through cloud services the target already trusts. 

None of it is sophisticated. All of it is optimized in the same direction, toward fewer observable steps and lower cost. 

That is where attacker effort is actually going, and it is the precise opposite of what an autonomous agent does. 

To understand why, follow the money. 

More Victims, Less Money 

Ransomware is not a technology showcase. It is a return-on-investment business, and as we wrote in January, “every decision, from choosing an exploit to selecting a target, focuses on maximizing Return on Investment and minimizing operational risk.” 

How that business finds victims has changed. The dominant route in is now mass exploitation of internet-facing devices: VPN appliances, firewalls, gateways left unpatched. 

Verizon’s 2026 Data Breach Investigations Report puts edge device exploitation at 22% of exploitation-driven breaches, up from roughly 3%, with vulnerability exploitation overtaking stolen credentials as the leading way in for the first time in the report’s nineteen-year history. 

That shift matters more than any single figure, because opportunistic targeting decouples victim selection from victim value.

If you attack whoever left an appliance unpatched, exposure becomes the selection criterion and ability to pay stops being one. 

The returns have followed. Payment rates have fallen four years running, to roughly 20% in the last quarter of 2025, with Verizon finding 64% of victims refusing outright, up from 50% in 2022. Chainalysis puts total ransomware payments at roughly $820 million in 2025, down 8% on a year that was itself down 35%.

And 2025 was simultaneously a record year for victims named on leak sites. More victims, less money. 

This is where agentic autonomy runs out of road. It is the purest form of opportunistic targeting available: a machine that tries everything against everyone. It pushes attackers further along the exact axis already producing lower returns.

Cheap compute does not fix a business problem whose binding constraint is the victim’s ability to pay. 

Marcus Hutchins, among the most consistent public skeptics of AI-malware claims, arrives at the same place from a different direction.

Speaking to SC Media UK, he argued that generative tools are not going to write the next WannaCry, that cybercrime remains a human-oriented field, and that AI-generated code is poor enough that the people shipping it often cannot tell it does not work.

The hype has costs beyond the discourse: one MIT paper claiming 80% of ransomware involved AI was withdrawn after criticism. 

None of this holds forever. Threat actors will adopt autonomous operation the moment it makes business sense, quickly and without sentiment, exactly as they adopted ransomware-as-a-service and double extortion when those started paying.

Watch the economics, not the capability announcements. 

The Part That Should Worry You 

Here is the uncomfortable half. The same volume that makes agentic autonomy unattractive to attackers is what makes it unmanageable for defenders. 

Hugging Face’s security stack detected the intrusion. Multiple layers fired. Detection was not the failure.

What failed was severity: the alert “failed to correctly raise the alert’s criticality and trigger the on-call team, costing precious time in the response.”

The stack saw it. It just could not rank it. 

Our research into AI-generated tooling named the deliberate version of this: exhausting defenders through automated volume rather than technical brilliance.

The Hugging Face incident shows the same effect with nobody intending it.

No one aimed this at that security team. The agent was simply loud. The defensive consequence is identical either way, which is the part worth sitting with: the noise floor rises whether or not anyone is aiming it at you. 

What This Means for Security Leaders 

There is a reflex answer to all of this, and it is usually offered by people whose product is detection: fight AI with AI. It is an appealing slogan and an answer to a question this incident did not ask. The stack detected the attack.

Analysis was not the constraint either; Hugging Face eventually built an AI-assisted pipeline to reconstruct what had happened, and it worked. What failed was prioritization, and more detection produces more to prioritize.

Matching machine speed on the analysis side of a volume problem does not resolve the volume. It scales it, and then asks a human to referee. 

The controls that actually closed this off were not clever. Hugging Face’s own remediation, published after a full investigation, is almost entirely preventive: tighter access boundaries, isolation between systems, credentials rotated, scope reduced.

One detection improvement appears in the list, and it is about raising the severity of signals they were already receiving. 

That is the general lesson, and it is not a new one. Preventive controls are volume-invariant. An access boundary does not get noisier as request rates climb, does not degrade under load, and does not need anyone awake at three in the morning.

Detection, correlation and triage all degrade as volume rises. If speed and volume really are about to increase, the controls that hold their value are the ones that never depended on someone reading an alert in time. 

None of which is an argument against detection, and we should be careful here, because the industry has spent a decade over-correcting in one direction.

As we put it in January, detection tools “are critical, they are not a silver bullet,” and “it is now time to complement these reactive measures with an effective prevention strategy.” The operative word is complement.

We have invested heavily in seeing attacks and comparatively little in making them impossible, and the balance is wrong. 

So the advice for security leaders is unglamorous, which is usually a good sign. Stay focused on what actually matters. Follow the practices you already know work.

Resist the pull of whichever capability is currently being sold as the answer, because the techniques in this incident were textbook, the defenses were understood years before it happened, and nothing about it required a new category of product.

What changes is how quickly the window closes between “we should get to that” and “someone already did.” 

Cheap automation raises the capability floor for unsophisticated attackers and does almost nothing for the ceiling, because the best operators were already quiet and deliberate and gain nothing from a machine that tries everything.

The organizations most exposed are the ones at the bottom of the maturity curve. For them, the fix is the work that has been on the list for years. 

You cannot buy your way out of a problem you configured your way into.