New benchmark RealCLI reveals sharp gaps in CLI agents’ collaboration and steerability

A new arXiv paper introduces RealCLI, a benchmark that unifies evaluation of task execution and human-agent collaboration quality for command-line interface (CLI) agents. Unlike prior benchmarks that assume fixed user instructions or focus solely on outcomes, RealCLI operationalizes collaboration across five dimensions, 22 sub-dimensions, and 51 interaction scenario templates.

The benchmark provides two evaluation modes: REALCLI-EXEC, which supplies complete task specifications upfront, and REALCLI-COLLAB, which simulates a scripted user that delivers progressive information, corrections, constraint revisions, and authorization decisions during execution.

Evaluations across frontier agents show significant capability gaps. The best configuration achieves an average score of 49.2%, and the peak performance on the joint collaboration score (Jcolla b) is only 37.6%. These results highlight a divergence between task effectiveness and collaboration quality, with models struggling particularly in steerability, trust calibration, and safety boundary management.

The authors describe RealCLI as a rigorous, reproducible foundation for measuring and advancing collaborative CLI agents, offering a path to more realistic and human-aligned evaluations of agentic systems.