Skip to content
Research · Nov 5, 2026

Verified telemetry cuts agent operational regret by 96.9% in controlled benchmark

Agent Harness Awareness supplies fresh, verified host state to models before consequential actions, reducing observable policy regret and boosting completion rates across three frontier models.

Trust79
HypeLow hype

1 source · cross-referenced

ShareXLinkedInEmail
TL;DR
  • Agent Harness Awareness introduces a trust-separated design where the host supplies a fresh, verified state snapshot to the model before a consequential action, addressing 'operational blindness' in tool-using agents.
  • In a 216-decision ablation across GPT-5.6 Sol, Claude Sonnet 5, and Gemini 3.7 Flash, verified tool-health state reduced mean observable policy regret by 96.9% (from 4,252 ms to 131 ms) and raised single-dispatch completion from 69.4% to 95.8%.
  • A 27-decision live execution replication with injected faults showed verified state reduced mean regret from 8,956 ms to 0 ms and achieved 9/9 successful completions versus 5/9 for the neutral control.
  • The approach retains host authority for measurement, enforcement, evaluation, and final dispatch while improving pre-dispatch operational choice.

Tool descriptions alone do not reveal whether an action is healthy or suitable at the moment of use, creating what the authors term 'operational blindness' in tool-using agents. Execution harnesses, however, often possess this information through monitors, traces, and recent failures, which the model does not have access to.

The proposed Agent Harness Awareness architecture addresses this gap with a trust-separated design: the host supplies a small, fresh, verified state snapshot to the model before a consequential action while retaining full authority for measurement, enforcement, evaluation, and final dispatch.

The authors evaluate the tools-and-network instance of this design using ToolRoute, a controlled isolation benchmark that holds task and candidate semantics constant while varying the operational information available to the model. In a 216-decision ablation across three frontier models—GPT-5.6 Sol, Claude Sonnet 5, and Gemini 3.7 Flash—verified tool-health state reduced mean observable policy regret by 96.9% compared to a same-shape neutral control (from 4,252 ms to 131 ms). Single-dispatch completion rates improved from 69.4% to 95.8%.

A separate 27-decision end-to-end replication with live local TCP/HTTP execution, OpenTelemetry client spans, and Toxiproxy-injected faults showed verified state reduced mean regret from 8,956 ms to 0 ms (zero excess observable operational cost). In this live setting, verified state achieved successful completion in all 9 model-by-fault cases, compared to 5 out of 9 for the neutral control.

The studies establish the tools-and-network instance of Agent Harness Awareness, demonstrating that selected host telemetry materially improves pre-dispatch operational choice. The authors note that runtime state, hardware pressure, economics, retry-aware recovery, and heterogeneous selection remain open empirical extensions of the same architecture.

Sources
  1. 01arXiv cs.AI — Agent Harness Awareness: Turning Verified Telemetry into Operational Control for Tool-Using Agents
Also on Research

Stories may contain errors. Dispatch is assembled with AI assistance and curated by human editors; despite the trust-score filter, mistakes happen. We correct publicly — every article links to its revision history. Nothing here is financial, legal, or medical advice. Verify before relying on any claim.

© 2026 Dispatch. No ads. No sponsorships. No paid placement. Reader-supported via Ko-fi.

Built by a person who cares about honest AI news.