AI advice reduced accuracy threefold while doubling confidence, study finds

A study by researchers from three French and Italian universities found that access to AI advice significantly degraded human performance on judgment tasks. Accuracy dropped from 27% to 9% when participants had AI assistance, while their reported confidence increased from 30% to 76%. The researchers used Step 3.5 Flash, a model that was typically incorrect on the test questions, to isolate the effect of AI availability from the reliability of the AI itself.

The study also measured a sharp decline in participants' willingness to acknowledge uncertainty. The share of responses indicating 'I don't know' fell from 44% to 3% when AI advice was available. Even when participants could have answered correctly without AI, some chose to consult the model and ended up providing incorrect answers. The authors argue this reflects a broader pattern: AI systems are often designed to provide answers rather than to say 'I don't know,' which may discourage users from recognizing the limits of their own knowledge.

Monetary incentives provided some mitigation. With incentives, willingness to admit ignorance rose from 3% to 8% and accuracy improved from 9% to 16%, though both remained well below baseline levels observed without AI (44% and 27%, respectively). The researchers note that this aligns with prior work describing the phenomenon as 'cognitive surrender'—a tendency to accept incorrect AI answers while feeling more confident than those working without AI.

The authors express particular concern about children, who may lack fully developed critical thinking skills and are growing up in environments where AI systems are ubiquitous. They point to recent product decisions, such as Google’s AI search overhaul that replaced links with AI-generated summaries, as examples of systems designed to provide answers rather than to acknowledge uncertainty.