Study finds metacognitive sensitivity in medical LLMs but highlights calibration gaps in high-uncertainty cases

Researchers developed a psychophysics-inspired clinical benchmark to test diagnostic choice and confidence behavior in a medical LLM, focusing on differentiating probable Alzheimer-type neurocognitive disorder (AT-NCD) from depression-related cognitive impairment (DRCI). The benchmark used 45 synthetic vignettes that varied evidence strength, conflicting evidence, and missing information, with each vignette presented under three prompt variants to yield 135 trials.

In a pilot run using gpt-4.1-nano, all trials produced valid structured outputs. Across forced-choice trials, the model achieved 93.5% diagnostic accuracy, a mean confidence of 78.4%, and an AUROC of 0.876. Confidence increased with evidence distance from the diagnostic boundary, decreased when information was missing, and remained higher on correct than incorrect trials after adjusting for evidence strength and prompt format.

The study reports partial metacognitive sensitivity, meaning the model’s confidence partially tracks evidence quality and uncertainty rather than being globally uninformative. However, errors were concentrated in moderate, conflicting AT-NCD cases, where the model shifted toward DRCI and retained more confidence than empirical accuracy justified.

The authors argue that confidence quality should be measured directly rather than inferred from benchmark accuracy or general model capability, and they propose the benchmark as a reproducible framework for evaluating evidence sensitivity, metacognitive sensitivity, and localized calibration failure in medical LLMs.