Apple researchers introduce inference-time feedback system for tool-calling agents

Apple Machine Learning Research has published a paper on a two-agent architecture that embeds correctness checking directly into agent execution. Rather than evaluating tool calls after they complete—a limitation that requires prompt tuning or retraining to address errors—the system deploys a secondary reviewer agent that validates provisional tool calls before they execute, enabling immediate mitigation of errors.

The core technical contribution is a framework that separates execution concerns from review. A primary agent selects tools and parameters; a reviewer agent audits those choices and flags errors in real time. The authors acknowledge this introduces new risk: the reviewer can itself generate errors while correcting others, a tradeoff rarely measured in prior multi-agent systems.

To quantify this dynamic, Apple researchers developed two metrics: helpfulness, which measures the percentage of base agent errors that feedback successfully corrects, and harmfulness, which measures the percentage of correct responses that feedback incorrectly flags or degrades. These metrics directly guide choices about which models and prompts to use in the reviewer role.

Testing on two benchmark suites—BFCL for single-turn interactions and τ2-Bench for stateful multi-turn scenarios—showed measurable gains: 5.5% improvement on irrelevance detection and 7.1% on multi-turn tasks. Critically, the choice of reviewer model mattered substantially. OpenAI's o3-mini reasoning model achieved a 3:1 benefit-to-risk ratio, compared to 2.1:1 for GPT-4o. Automated prompt optimization added another 1.5–2.8% of improvement.

The architecture's modularity means the base agent does not require retraining when improving the reviewer. This decoupling is practical for deployed systems where base models are fixed or expensive to retrain, allowing continued performance gains through reviewer refinement alone.