Anthropic conjures another threat that doesn't exist yet to stoke fears of AI apocalypse

Nothing sells AI products as well as fear. Using new research, Anthropic claims that self-replicating AI worms could be just months away from spreading across vulnerable IoT devices, which would be a problem if it came true.

The frontier lab published a research report Monday suggesting that AI models could eventually self-replicate without relying on datacenter hardware, with a 9-billion-parameter model successfully setting up inference on eight of the nine hardware types Anthropic tested.

“A replicator like this would be challenging to shut down, since there’s no centralized point to shut off,” Anthropic warned, adding that such an AI worm could also suss out the workings of unfamiliar devices, potentially making it more adaptable than conventional botnets like Mirai, Kimwolf (the representative IoT botnet threat model used in the study), and others.

Anthropic’s researchers managed to get a distributed instance of Gemma4-26B-A4B running across 15 generic Android set-top streaming boxes - a common target of traditional botnet malware due to their lax security. Once up and running, the distributed Gemma4 model was able to conduct inference tasks at a blistering 0.39 tokens per second. 

That, says Anthropic, proves that AI models can conduct inference with their various layers distributed across cheap hardware, but it doesn't prove such a model could self-replicate. For that, the researchers turned to Z.ai’s GLM 5.2. 

GLM was used to propagate a copy of Llama3.1-8B across eight Google Cloud instances running an Android 15 emulator in a virtual private cloud, which Anthropic said it was able to do with a high degree of success, with the model successfully spreading to all eight nodes in four out of five trials, and seven of eight in the other. 

“It appears that GLM-5.2 is able to propagate Llama3.1-8B well, which suggests that future small models of the same capability level could also complete the network enumeration, payload delivery, and inference standup steps of the replication process,” Anthropic explained. 

Sanity check: This isn’t going to end the world, either

Frontier AI labs have become adept at publishing findings that portend doom that only they can prevent. Doing more than a cursory reading of this report suggests that this threat is just as deserving of a big, honking asterisk as the rest.

Take, for example, what is needed for an AI model to exploit vulnerabilities and spread to various systems. One would think that such an element would be an important part of demonstrating the danger of a self-replicating AI botnet, but Anthropic decided it didn’t need to worry about that for this study. 

“We largely ignore the cyber exploit element of the replication process, since earlier papers CyberGym, ExploitBench, and adaptive worms already show that small models are capable of cyber exploits, especially when given the right tools,” Anthropic explained in the writeup.

In other words, the replication these models managed to achieve was done on nodes that were custom-built for this project and deliberately left open to attack, although the draggingly slow inference demonstration ran on real, inexpensive Android TV boxes. As Anthropic notes, AI models can also benefit from being given the right tools to turn them into capable cyberattack agents. There was a custom-built harness called Memento involved in this scenario too. 

Are IoT nodes that get hijacked by botnets famously insecure? Yes, but it doesn’t take much to prevent the most common ingress routes – that’s just good security hygiene that has nothing to do with the involvement of AI. Anthropic acknowledged that its deliberately simplified exploitation tests provided “an upper bound on real-world success.”

Then there’s the aforementioned speed. Anthropic says such AI worms would be challenging to shut down, but estimates that they might take days to double in size. When Mirai appeared on the scene in 2016, it was doubling in around an hour. 

Anthropic points out that speed is a factor in detection and mitigation, too. AI models are not simple malware packages: They’re massive things that would eat up tons of bandwidth and quickly point to their sources. As an AI botnet grew, that traffic could become more noticeable, making infected devices easier to detect.

Finally, end-to-end replication may have been successful with GLM-5.2, but that’s it. Anthropic tested other open-weight models, and they all “fail[ed] reliably,” according to the report. Anthropic didn’t mention how many models it tested. The report also mentions that GLM’s unique success has another important condition: It assumes its capabilities “could be packed into Llama3.1-8B,” stacking another assumption onto this latest house of doomsaying AI cards.

“Models of 30B parameters and under can already complete many of the individual steps for replication today, though they make mistakes when composing them end to end,” Anthropic said, while noting that GLM-5.2 was capable of end-to-end replication, conveniently leaving out all the important caveats. 

Anthropic further admits that it has no idea "exactly what form autonomous AI replication will take,” or when it might arise, if ever. “If small models improve at their historical rate, they’ll match this capability level by the end of this year,” the company added – again imposing an essential condition on this supposed threat. 

“As a prediction that goes beyond our experiments, we expect a future awash with agents running on all types of hardware, including some that aren’t under human control,” the company said at the end of its report. “We should aim to harden our systems to make it difficult for such replicators to spread, but also prepare to deal with a world where they are here to stay.”

Here to stay for a given value of not existing yet, of course. Yet again, a frontier lab is warning us of a threat that doesn’t exist yet – at least in this case, basic security measures could make life harder for any future AI-powered botnet.  ®