ClinLens benchmark exposes gap between executable and correct clinical coding agents

Researchers introduced ClinLens, a benchmark designed to evaluate clinical data-science agents on heterogeneous longitudinal records, including structured electronic health records, clinical notes, electrocardiograms, chest radiographs, and echocardiograms drawn from five linked MIMIC resources.

The benchmark comprises 200 executable tasks organized under a 4 x 5 taxonomy that crosses four patient-time scopes with five analysis capabilities, enabling standardized evaluation of agents' ability to handle multimodal and temporal clinical data.

Evaluation uses a program-first reverse synthesis approach, where each task is paired with an evaluator-private reference workflow and checked for required artifacts, cohort and temporal semantics, and the final answer, ensuring auditable and reproducible analyses.

On a fixed 126-task suite, the strongest of 24 standardized model-scaffold configurations achieved 56.3% scope-macro STRICTPASS while maintaining 100% EXECSUCCESS, indicating that while all submissions executed without error, only about half met correctness criteria across patient-time scopes.

By comparison, a separately configured coding agent solved 83 of the 126 tasks, while five biomedical systems adapted to GPT-4o-mini reached at most 2.9% scope-macro STRICTPASS, underscoring a substantial performance gap between general-purpose coding agents and domain-adapted biomedical systems.

The authors note that these results expose a meaningful divide between runnable submissions and correct clinical analyses, emphasizing the need for benchmarks that stress both executability and clinical validity in longitudinal multimodal settings.