Researchers propose synthetic customer agents to validate LLM chatbots in banking

Researchers from multiple institutions have published a methodology for validating large-scale LLM-based chatbots using synthetic customer agents (SCAs) as high-fidelity digital twins grounded in real transactional and conversational data. The work, presented in an arXiv preprint, aims to address the challenge of scalable and cost-effective validation in regulated domains such as banking, where safety and compliance are critical.

The authors introduce a two-part contribution: first, a methodology to create SCAs that simulate diverse customer profiles and interaction styles with controllable behavioral conditioning; second, an SCA-based validation framework that integrates automated LLM-as-a-Judge evaluation, human expert testing, and adversarial probing. The framework is designed to assess robustness across emotional states, demographic groups, and linguistic factors.

The evaluation reported in the paper claims high semantic alignment between SCAs and real customers, low hallucination rates in the synthetic interactions, and successful reproduction of personality traits with controllable interventions. The authors state that the approach was used to validate a customer-facing chatbot at a leading UK bank, providing financial institutions with a scalable pathway toward regulatory compliance.

The preprint is authored by researchers affiliated with multiple institutions and was submitted to arXiv on May 15, 2026. It is categorized under Computation and Language (cs.CL) and Artificial Intelligence (cs.AI).