Report: US almost boarded Chinese ship over hallucinated AI arms report
A SOCOM analyst's AI-assisted intelligence report falsely claimed a Chinese ship carried nuclear components, nearly triggering a US military boarding operation.
A CNN report says a US Special Operations Command analyst used a chatbot to analyze intelligence on a Chinese vessel's manifest, and the chatbot fused open-source and signals intelligence while misidentifying the cargo as nuclear arms program components. The US military prepared an interception with air support before officials discovered the error; one source said it 'almost started a war.' The episode follows the Pentagon's January 'AI acceleration strategy' promoting AI use across federated defense systems.
- SOCOM analyst used a chatbot that misidentified a Chinese ship's cargo as nuclear program components
- US military prepared a boarding operation with air support before the hallucinated report was caught
- Chatbot fused open-source intelligence with secret signals intelligence into the flawed report
- DoD's January AI acceleration strategy pushes AI exploitation across federated mission systems
Full article339 words · extracted from arstechnica.com · click to collapse
The US narrowly avoided boarding a Chinese ship based on an “entirely false” US intelligence report generated with the help of AI tools, according to a CNN report.
That erroneous report, submitted by a US Special Operations Command analyst, erroneously suggested the Chinese ship was transporting nuclear arms program components through the Middle East, according to “four sources familiar with the episode” cited by CNN. The US military was preparing to intercept and board the ship, with air support, before officials discovered a chatbot used in generating the report had “inaccurately identified the material the ship was carrying.”
One source told CNN the AI-powered fiasco “almost started a war.”
What’s the worst that can happen?
CNN’s report said the analyst in question used a chatbot to analyze intelligence reports regarding the Chinese ship’s manifest, leading to the near-disastrous result. That chatbot “fused together open-source intelligence with secret signals intelligence in government holdings,” and that information was packaged into an intelligence report that almost set off a disastrous chain of events, according to CNN.
The near-miss is one of the more potent and consequential instances of a hallucinating AI ruining the reliability of a professional report. Since “hallucinating” became the Cambridge Dictionary’s word of the year in 2023, we’ve seen prominent examples of non-fiction authors, journalists, academic researchers, judges, doctors, police departments, corporate call centers, and more getting taken in by AI tools that simply make something up when their training data doesn’t provide sufficient context. And despite some adorable attempts at “do not hallucinate” prompts, some researchers suggest that it may be impossible to prevent LLMs from hallucinating altogether.
One would hope the US military would be aware of these kinds of problems when relying on AI for analysis of intelligence reports. But the Department of Defense in January rolled out an “AI acceleration strategy” that sought to “make all appropriate data available across federated IT systems for AI exploitation, including mission systems across every service and component.”