Researchers propose ProvenanceGuard, a source-aware verification layer for MCP-based agents to reduce cross-source conflation errors

A new verification layer called ProvenanceGuard targets a subtle failure mode in tool-using LLM agents: cross-source conflation, where a claim is supported somewhere in the evidence but attributed to the wrong source.

ProvenanceGuard operates as a post-generation step on top of a black-box MCP agent, reading the captured MCP trace—including tool outputs and their source IDs—without retraining the agent.

The system breaks an answer into specific claims, finds the most relevant source for each, checks whether that source supports the claim, compares the source with the one the answer names or implies, and then emits both per-claim source verdicts and a global allow or block decision.

In experiments using 281 real traces from a medical agent that queried patient records, research articles, and other tools, human experts set aside 40 answers containing 361 claims for evaluation. Experts judged 139 claims unsupported; ProvenanceGuard blocked 138 of them and allowed one through.

ProvenanceGuard also held 67 claims that experts considered supported, reflecting a conservative policy that favors second looks over letting unsupported claims pass. For claims with an identifiable source, it selected the correct source about 86% of the time in this test.

Across the same held-out claims, ProvenanceGuard’s blocking performance (F1 0.802) matched or exceeded source-blind baselines such as MiniCheck (0.783), RAGAS Faithfulness (0.758), AlignScore (0.662), and SummaC-ZS (0.436).

In a separate test with several similar sources, ProvenanceGuard achieved 0.846 F1 for blocking decisions but identified the exact source correctly in 50.3% of claims, highlighting room for improvement in distinguishing closely related sources.

The authors note that ProvenanceGuard’s local-model setup—using MiniLM, a DeBERTa NLI verifier, and a local LLM—was chosen for controlled, offline evaluation and is not a requirement; the same verification flow can be adapted to hosted models with appropriate testing and calibration.