Paper proposes Calibrated User Embeddings to improve multi-turn AI agent benchmarking

A new arXiv preprint introduces Calibrated User Embeddings (CUE), a framework designed to improve the ecological validity of user simulators used in multi-turn AI agent benchmarks. The authors argue that existing simulators often lack outcome calibration, meaning they do not align with observed success rates and failure patterns when real users interact with the same agent. This misalignment can lead to misleading evaluations of AI systems.

The CUE framework encodes observed user sessions into continuous representations and decodes them into persona commands that steer large language models (LLMs) to act as user simulators without additional training. This approach enables two evaluation modes: user-conditioned replay of past sessions and sampling novel personas for the same tasks.

On the τ²-Bench benchmark, CUEd simulators commit fewer simulator-attributed errors and more faithfully reproduce real-user agent failure modes, aggregate success rates, and outcomes for specific task-user pairs compared to other persona-based simulation methods. The authors report that these gains are achieved while maintaining competitive user fidelity as measured by established metrics.

The paper demonstrates that CUEd simulators, after being trained primarily on customer support interactions, generalize to other domains such as document creation, math tutoring, and casual conversation. Additionally, the method remains effective across different simulator LLMs without requiring CUE retraining, suggesting broad applicability and efficiency.