LLM watermarks degrade medical text quality across multiple failure modes, study finds

A new arXiv preprint introduces the first rigorous evaluation of how LLM watermarking schemes affect medical text generation, benchmarking five watermarking methods across 11 large language models and seven vision-language models on clinical reasoning tasks. The study spans both unimodal and multimodal settings, reflecting real-world clinical workflows where LLMs may process text alongside images or other data.

The authors report that watermarking can induce substantial degradation across multiple failure modes, including lexical corruption, hallucinated terminology, and amplified misattribution or omission of image findings. These errors are particularly concerning in medicine, where small token-level perturbations can result in significant semantic changes or misdiagnoses.

To assess watermark-induced failures, the team developed a human-expert-validated pipeline for auditing medical reasoning quality, terminological precision, and induced hallucinations. This pipeline provides a structured approach to detect subtle but clinically consequential errors that aggregate metrics may overlook.

The findings emphasize that the absence of domain-specific analyses can systematically obscure practical watermark-induced degradations. Current benchmarks, designed for general-purpose text, fail to capture the unique failure modes of clinical text, making domain-specific evaluation a prerequisite for safe deployment in medical settings.