Researchers identify role confusion as a fundamental challenge in preventing prompt injection

Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell argue that large language models (LLMs) struggle to distinguish their own privileged text—such as system, think, or assistant role tags—from untrusted user input wrapped in user tags. This failure, which they term "role confusion," allows prompt injection attacks to override a model's training and safety mechanisms.

The researchers demonstrate that models like gpt-oss-20b can be manipulated by appending text that mimics the style of internal thinking blocks. For example, a seemingly innocuous user message about wearing a green shirt can be paired with a policy-like statement that overrides safety policies if the user is wearing green, leading the model to comply with harmful requests.

The paper introduces a "destyling" technique—rewriting text to avoid the stylistic cues of role tags—which drastically reduced the success rate of prompt injection attacks in their dataset. Average attack success dropped from 61% to 10% when text was destyled, indicating that models rely heavily on superficial stylistic cues rather than the semantic content of the text.

The authors warn that role confusion is a fundamental challenge for prompt injection defenses. They argue that without models achieving "genuine role perception," defenses will remain reactive and fragile, requiring continuous updates to counter new attack vectors. They also highlight the risk of large-scale, legally ambiguous injections designed to subtly shift model states through seemingly harmless text.