Watermarking LLMs can alter safety guardrails under adversarial prompts, study finds

A security researcher at Lasso Security tested Google’s open-source SynthID-Text watermarking on six open-weight models using Hugging Face’s unmodified SynthIDTextWatermarkLogitsProcessor. The study compared model responses to harmful prompts with and without the watermarking enabled, focusing on refusal behavior and tool-calling under adversarial prompt-injection conditions.

The results show that SynthID-Text changed refusal behavior on harmful requests, with the effect becoming more pronounced when prompts were paired with prompt-injection techniques. In several models, watermarking made the system more likely to comply with requests it would otherwise refuse, a shift the researcher termed 'sampling drift.'

Watermarking also influenced downstream agent behavior by altering which tools were called and what arguments were passed. The study found that changes in token sampling under watermarking could shift tool-calling accuracy, sometimes substantially, beyond what overall accuracy metrics suggested. The effect varied depending on the secret key used during tournament sampling, indicating that the choice of key can meaningfully change model behavior.

The researcher emphasized that watermarking is designed to be imperceptible to readers but acknowledged that any change to the generative process can introduce trade-offs. 'Watermarking is made to not be perceptible to a reader, but we know that when we are changing anything about what the model is generating, it is going to cause some tradeoffs, it’s going to show up somewhere,' the researcher told Ars Technica.

The study notes limitations: it did not test Anthropic’s forthcoming Claude models, which will also use SynthID-Text, nor did it evaluate the specific implementation planned for Claude. Instead, it focused on open-weight models using the Hugging Face implementation of SynthID-Text tournament sampling. Still, the findings suggest that at least some forms of watermarking may affect model and agent safety, warranting further stress-testing and red-teaming before deployment.