Medical LLM performance hinges on data mix, study finds

A new arXiv preprint reports token-matched experiments that isolate how didactic versus clinical data shapes medical large language models. The authors vary the didactic-to-clinical ratio and measure effects on knowledge-intensive and clinic-oriented tasks. They find clinical data improves performance on clinic-oriented tasks without sacrificing performance on knowledge-intensive tasks, whereas didactic data primarily enhances knowledge recall. Error analyses reveal a knowing-doing gap: gains in knowledge do not reliably translate to better clinical reasoning.

The study also finds that modest amounts of clinical data yield most of the gains on EHR-grounded tasks, and that the optimal data mixture depends on the downstream task’s demands for knowledge and clinical reasoning. The authors conclude that medical LLM data curation should be application-driven, favoring higher proportions of clinical data for reasoning-intensive use cases.

The paper, titled 'Didactic knowledge or Clinical Cases? How Data Types Shape Medical Large Language Models,' is slated for an oral presentation at NLPCC 2026. The authors include Yuzheng Fan, Haochun Wang, Sendong Zhao, Xiao Han, Ming Ma, and Bing Qin.