Provenance-based framework reduces LLM agent misalignment errors by up to 96%

Researchers propose ProvenanceGuard, a provenance-based framework that formalizes misalignment detection as verifying whether a proposed tool call is supported by traceable evidence in the agent’s context.

The approach introduces a multi-stage pipeline that analyzes an agent’s proposed action for three types of misalignment before the tool is executed, allowing the action only when it aligns with the user’s input query.

In evaluations on Agent-SafetyBench and WorkBench across ten backbone LLMs, ProvenanceGuard reduced error rates on misaligned traces from 42.9% to 1.8% on Agent-SafetyBench and from 32.1% to 17.3% on WorkBench compared to LLM-as-a-judge baselines.

The method also lowered intervention burden on task-successful traces from 30.5% to 12.8% and showed no statistically significant increase in unnecessary interventions on aligned traces, indicating improved precision without added overhead on correct actions.

The authors argue that provenance-based reasoning provides a more systematic and auditable alternative to LLM-as-a-judge paradigms, which often produce inconsistent or hard-to-audit judgments.