Unified text serialization enables LLMs to match or exceed task-specific multimodal baselines in clinical prediction tasks

Electronic health records (EHRs) typically combine free-text clinical narratives with structured measurements such as vital signs, laboratory values, and comorbidities. Most clinical prediction systems address this multimodal data by designing task-specific fusion architectures, which pair dedicated encoders for each modality with learned combination mechanisms that must be re-engineered for every new task and clinical setting.

The authors propose an alternative approach: convert all patient data, regardless of modality, into a single natural language sequence and fine-tune a pretrained language model end-to-end without architectural modifications for fusion. This unified textual serialization method aims to simplify the development pipeline by removing the need for bespoke multimodal fusion designs.

The approach was evaluated across three clinically distinct prediction tasks. First, in-hospital mortality prediction was assessed using the MIMIC-III dataset. Second, graft failure prediction was evaluated using longitudinal data from a German transplant center, comparing against a gradient boosting model currently used in clinical practice for post-transplant patient management. Third, emergency triage classification was tested using ambulance records.

The study compared encoder-based (ModernBERT) and decoder-based (Llama 3.1, Gemma, DeepSeek-R1-Qwen, Qwen3) fine-tuning strategies against established multimodal baselines. Across all three tasks, the unified textual serialization method matched or exceeded the performance of task-specific multimodal baselines. Notably, it outperformed the clinically deployed gradient boosting system on graft failure prediction, indicating potential real-world utility.

The results suggest that a single serialization-based paradigm, without bespoke fusion architectures, is sufficient for multimodal clinical prediction. This approach substantially reduces system complexity while matching or exceeding the performance of specialized, task-specific designs.