Skip to content
Safety · Nov 9, 2026

Mechanistic study finds multiple-choice format suppresses model refusals on safety benchmarks

Tuned lens analysis shows models commit to answer-letter tokens early in MCQs, pulling activations away from refusal directions and reducing safety behavior measurement.

Trust79
HypeLow hype

1 source · cross-referenced

ShareXLinkedInEmail
TL;DR
  • Safety benchmarks often use multiple-choice questions (MCQs), but prior work showed MCQs reduce refusal rates in models.
  • Researchers used the tuned lens technique to show models commit to answer-letter tokens early in MCQs, unlike gradual refusal buildup in open-ended formats.
  • Activations in MCQ format lie roughly 3x further from refusal direction compared to open-ended formats.
  • Ablating attention heads carrying MCQ answer-selection increases refusal, indicating competition between answer selection and refusal mechanisms.
  • Findings suggest MCQ safety scores primarily measure knowledge, not safety mechanisms.

Safety evaluations of large language models (LLMs) frequently rely on multiple-choice question (MCQ) formats, but prior studies have observed that this format reduces refusal rates compared to open-ended prompts. A new arXiv preprint provides a mechanistic explanation for this discrepancy using the Weapons of Mass Destruction Proxy (WMDP) dataset. The authors analyze two models—Llama-3.1-8B and Qwen3-8B—and find that in MCQ settings, models commit to answer-letter tokens early in processing, contrasting with the gradual refusal buildup seen in open-ended formats.

Using the tuned lens technique, the researchers show that activations during MCQ processing lie roughly three times further from the refusal direction than those in open-ended formats. This suggests that the MCQ format pulls model activations away from refusal-related directions, thereby suppressing refusal behaviors. Further evidence comes from ablation experiments: when attention heads responsible for carrying MCQ answer-selection information are ablated, refusal rates increase. The authors interpret this as direct evidence that answer-selection in MCQs competes with refusal mechanisms.

The study concludes that MCQ-based safety scores may primarily reflect knowledge capabilities rather than the effectiveness of safety mechanisms. This calls into question the validity of using MCQ formats as proxies for safety behavior in benchmarking, especially where refusal is a critical component of safety evaluation.

Sources
  1. 01arXiv cs.CL — Forcing a Letter Suppresses Refusal: Mechanistic Evidence from WMDP
Also on Safety

Stories may contain errors. Dispatch is assembled with AI assistance and curated by human editors; despite the trust-score filter, mistakes happen. We correct publicly — every article links to its revision history. Nothing here is financial, legal, or medical advice. Verify before relying on any claim.

© 2026 Dispatch. No ads. No sponsorships. No paid placement. Reader-supported via Ko-fi.

Built by a person who cares about honest AI news.