Paper argues role tags in LLMs are not robust to prompt injection and calls for stronger role perception

A recently highlighted paper titled 'Prompt Injection as Role Confusion' argues that large language models (LLMs) do not internally enforce the role boundaries implied by formatting tags such as 'user', 'assistant', or 'think'. Instead, models learn to recognize stylistic patterns associated with these roles, which can be mimicked or manipulated by adversaries.

The authors contend that this role confusion is directly linked to prompt injection vulnerabilities, where malicious inputs can coax models into disregarding their intended instructions. They warn that unless LLMs develop 'genuine role perception'—a capacity to reliably distinguish roles at an architectural level—defenses will remain a 'perpetual whack-a-mole game'.

The paper further cautions that the continuous nature of role boundaries in LLMs opens the door to injections that subtly shift model states through seemingly innocuous text, potentially at scale and within legal frameworks.

Commenters on the topic emphasize that this framing reframes prompt injection not just as a surface-level input sanitization problem, but as a foundational architectural and alignment challenge. Some argue that current guardrails—whether at input or output—can be bypassed via obfuscation or encryption, reinforcing the need for deeper architectural changes rather than superficial fixes.

The authors also point to empirical evidence, such as 'CoT Forgery', where user-supplied text mimics the style of a model's internal 'think' role, leading the model to accept forged reasoning as its own. They report success rates near 60% across tested LLMs, underscoring the breadth of the issue.