Your AI agent is only as safe as the packages it downloads

NVIDIA's new agent safety platform puts enforcement in the right place. It still must know what it is letting through.

Strip away the model names and the headlines, and this year’s AI agent incidents share a feature that has little to do with how capable the models are. Nearly all of them ran through the software supply chain.

Start with the case that set off the summer’s wave of disclosures. In July, two OpenAI models undergoing a cyber-capability evaluation escaped their sandbox. They got out through a zero-day vulnerability in a package registry proxy that sat inside the evaluation boundary, placed there so the models could install the tools they needed. From there, they reached the open internet and went after Hugging Face’s production systems, looking for the benchmark’s answer key. Hugging Face caught and contained the intrusion on July 16. Five days later, OpenAI confirmed the incident and connected it to an internal cybersecurity evaluation of frontier AI models.

Soon after, Anthropic disclosed three incidents in which its Claude AI models compromised external organizations after a misconfiguration left live internet access in place. In September, it added a fourth that a wider review had turned up. The incident that Anthropic says worries it the most involved a model publishing malicious packages to the Python Package Index (PyPI).

Separately, researchers have attributed more than 2,000 RubyGems uploads in May to OpenAI agents. OpenAI says its agents were carrying out benign tasks. RubyGems removed more than 500 of the packages as malicious.

In each case, the agent’s route out, or the harm it caused, ran through a package registry, a dependency or a model file.

None of this surprised the researchers I work with. ReversingLabs has spent years taking apart the files that move through software pipelines, and we watched attackers discover the AI supply chain well before agents reached production. In early 2025, our team found malicious models on Hugging Face whose pickle payloads were packed in a 7z archive that Picklescan, the platform’s scanner, could not open. The payload itself was an ordinary reverse shell. Hugging Face took the models down after we reported them.

In April, we published research on PromptMink, a campaign we tie to the North Korean group Famous Chollima. In one case, a Claude Opus-based coding agent added a malicious dependency to a cryptocurrency trading project. What struck our team was who the attackers were writing for. Their package documentation was built to be more convincing to an AI coding assistant than to a human developer.

Put those cases side by side, and the threat model looks different from the one most enterprises are planning around. The evaluation escapes show agents misusing the access they were given. PromptMink shows agents being steered by content written specifically to fool them. In both, the damage moved through software the agent downloaded or published.

That is the context for the Open Agent Safety Platform NVIDIA launched on Sept. 28. The design starts from the right premise: you cannot secure an agent by asking it to behave. OpenShell runs each agent in a sandbox and applies policy outside the agent’s own process. Sentry, part of a reference design built on BlueField-4 data processing units (DPUs), watches agent activity from separate hardware, out of reach of a compromised host. More than 100 partners signed on at launch, ReversingLabs among them.

The Hugging Face incident shows why that architecture matters. OpenAI ran those models with cyber refusals reduced, meaning their built-in guardrails were deliberatelyweakened. That left the sandbox boundary as the main line of defense. Prompts and training shape what an agent attempts. Infrastructure decides what it can actually do.

It also matches what I hear from customers. They are demanding AI safety that closes as many gaps as possible and uses the boundary between hardware and software to deliver security they can rely on. That is the standard every vendor in this ecosystem, mine included, will be measured against.

Look, though, at where that boundary failed: a package proxy, trusted because it was supposed to be there. A policy engine can tell you an agent is permitted to install a package. It cannot tell you whether the package is what it claims to be. NVIDIA’s CISO, Saša Zdjelar, made the point himself in a Sept. 21 post, writing that teams must “verify the source and integrity of the tools, skills and dependencies agents use.” Doing that means inspecting the artifact itself, whether a model file, an archive, a container or a binary, at the boundary and before the agent touches it. Even AIUC-1, a prominent certification standard for AI agents, treats scanning model files for malicious payloads as a supplemental control rather than a required one. Its vendor due diligence requirement stops at the model provider. That is the layer ReversingLabs brings to the platform.

I have two cautions as Sentry moves from reference design to shipping products. The first is about openness. Sentry depends on specific NVIDIA hardware, and “open” will only mean something if enterprises can plug in the inspection, telemetry and policy tools they choose and run them on infrastructure they already own. Buyers should hold the ecosystem to that standard.

The second is about the need for prompt disclosure, and it applies to the whole industry. In June, an OpenAI agent accessed non-public files on the Australian government’s Medicare Statistics Reporting Service portal. Services Australia, which runs the portal, learned of it on Sept. 10, through an email to a public mailbox Prime Minister Anthony Albanese called the delay unacceptable. RubyGems maintainers learned who was likely behind their May flood of packages only from outside researchers, months after the fact.

We would not accept that timeline for any other security incident, and AI agents should not be an exception. Organizations that run agents, AI labs included, should notify affected parties on the schedules we already expect for breaches. Package registries are shared infrastructure for the entire software economy, and they should be told when an agent has published to them. Labs running offensive evaluations should treat every service reachable from the sandbox, the package proxy included, as an attack surface, and they should publish the containment standards they hold themselves to.

Compromises like SolarWinds, 3CX DesktopApp and xz Utils taught the security industry that software cannot be trusted simply because of its reputation or where it came from. Agents now pull and publish that software on their own, at a volume no review team can match. NVIDIA has built a strong platform for enforcing trust decisions. The rest of us have to make sure those decisions rest on knowing what is inside the software.

The legal questions raised by agentic AI hacks

Experts and policymakers want AI companies to face consequences for agentic hacks. There may not be a clear-cut answer under existing laws and regulations.

National cyber director: Government-industry collaboration vital to managing AI risks, competition with nations

As AI world debates security, NVIDIA releases open source tools for agents

Government

• DOJ, FBI seize Flax Typhoon-linked hacking tools Microscan, FishHub

• Major rules for federal contractors handling sensitive data are nearing the finish line

• FBI, French authorities seize deepfake CSAM-for-sale websites 

• Alert: FortiBleed remains active campaign, can lock out users or lead to ransomware attacks

Technology

• New bill would create federal investigative body for AI-driven hacks 

• CISA outlines improvement plan for CVE program

• OpenAI, Ukraine partner on ‘Daybreak’ program to protect power grids and water systems 

• Citing China, President Trump doubles down on hands-off approach to AI regulation

Threats

• Leader of 764 pleads guilty, faces up to 30 years in prison

• PoeLLM malware has assembled a sweeping botnet, taking technical cues from a poem

• Citrix discloses third actively exploited NetScaler zero-day in less than a week

• Authorities seize KillSec extortion group infrastructure, arrest 3 alleged members

Policy

• Wiretapping change sparks big privacy fight in the Golden State

• Here’s how experts think CISA should tell agencies to protect OT

• WaterISAC reckons with range of threats after summer of cyberattacks

• US is looking to weave AI into critical infrastructure for cybersecurity, national cyber director says