AuditKIT proposed as unified evaluation framework for LLMs and agentic systems

Researchers from Lexsi Labs propose AuditKIT, a source-available evaluation library designed to unify how large language models (LLMs) and agentic systems are benchmarked, judged, and compared. The work argues that current evaluation practices are fragmented: differences in interfaces, metric semantics, and trace formats complicate comparisons and can obscure failures in agentic workflows.

AuditKIT introduces a standardized result representation that records benchmark accuracy, judge-based assessment, runtime performance, tool-use behavior, and retrieval quality in a single format. The library includes direction-aware metric contracts, configuration fingerprints, and extensible metrics and backends to support diverse evaluation needs.

For agentic systems, AuditKIT provides turn-aware tool-call matching, distinguishes between parallel and sequential execution, and separates verified task outcomes from trajectory-level diagnostics. The framework explicitly marks missing or unobservable evidence as unscored rather than counting it as a failure, aiming to reduce false negatives in evaluation.

For retrieval-augmented generation (RAG), AuditKIT combines ranked retrieval metrics, judged grounding, and deterministic evidence checks to provide a more comprehensive assessment of system performance.

The authors present seven case studies spanning model comparison, serving runtimes, retrieval, tool calling, and interoperability. These studies illustrate how factors such as answer extraction methods, runtime behavior, and metric selection can materially change evaluation outcomes, underscoring the need for standardized evaluation practices.

The paper is published on arXiv as version 1 on October 15, 2026, with 25 pages, 2 figures, and 13 tables. The code repository is available via a provided URL, and the authors include Shreeyans Arora, Utsav Avaiya, Zera Lyngkhoi, Hem Gosalia, Ram Mohan Rao Kadiyala, Vinay Kumar Sankarapu, and Pratinav Seth.