Study finds 91 silent failures in agent-tool interactions across 15 scientific tools

A new arXiv preprint introduces a study on silent failures in agent-tool interactions, focusing on biology agentic workflows within the ToolUniverse environment. The research defines silent failures as cases where a tool invocation appears successful but returns incomplete or missing information via API or wrapper, without any notification to the user or agent.

The authors developed an audit mechanism to identify these failures, examining 15 scientific tools and their associated API and tool documentation. They characterized failures across seven loci in the interaction chain and manually validated 91 silent failures after an initial LLM-based candidate discovery and automated testing phase.

The most frequent failure types were missing data or fields (unspecified counts) and inconsistencies in search, filtering, or ranking criteria. Most of the observed failures originated in the API layer (51) or wrapper layer (25), with the potential for silent failure amplification as errors propagate downstream into downstream outputs.

The results indicate that silent failures begin upstream and can lead to apparently valid but incorrect scientific outputs. To address this, the authors propose a concept of contextual reliability and recommend mechanisms for testing, disclosing, monitoring, and measuring such failures across the agent-tool interaction pipeline.