AI hallucination in U.S. military intelligence report narrowly avoided armed conflict during Iran war

A U.S. military AI chatbot produced false intelligence during the war with Iran that nearly led to an armed operation against a Chinese vessel, according to a report by CNN and coverage from TechCrunch. The erroneous intelligence originated when a Special Operations Command analyst queried an AI chatbot to synthesize open-source data with classified signals intelligence. The chatbot misidentified the vessel’s cargo manifest, falsely claiming it carried components for a nuclear weapons program.

The analyst then used the AI tool a second time to format the incorrect findings into an official-looking summary, which was circulated across military command channels. The operation was aborted only after U.S. officials discovered the intelligence was hallucinated by the AI system. The near-miss occurred as the Pentagon accelerates AI integration to speed up decision-making and maintain a strategic edge over China.

The incident underscores concerns among military officials and external experts about the risks of AI errors propagating through decision-making chains before being identified. ‘It’s important for service members to understand the uncertainty inherent to LLMs,’ said Jake Steckler, research scholar at GovAI and veteran U.S. Army officer, in a written response to TechCrunch. ‘But it’s especially critical for any decisions that could lead to use of force, like targeting, intelligence analysis, or operational planning. There are life and death consequences for those decisions.’

Steckler emphasized that the incident should prompt the addition of more safeguards rather than discouraging AI adoption altogether. ‘These tools can be useful in the right contexts and with the right safeguards in place,’ he said. ‘But prioritizing adoption speed over all else will likely lead to incidents that only make service members lose trust in these systems, which ultimately is only going to slow adoption.’