Paper identifies activation bottlenecks as cause of hidden-constraint failures in large language models

Researchers from Carnegie Mellon University propose a framework called conditional constraint activation to distinguish between a model’s internal encoding of a constraint and its actual routing into the decision process. They argue that aggregate accuracy metrics can obscure genuine constraint inference by conflating it with conservative defaulting behavior.

In experiments spanning 14 models, the team introduces a quartet diagnostic to isolate two distinct failure modes tied to hidden constraints. Using probes on two open-weight models, they demonstrate that constraints can be decoded from internal activations with accuracy above 88%, indicating that the knowledge is present but not always used.

Activation patching experiments show partial repair: one failure mode improves by +6.4 nats, while the other remains unchanged (-0.07 nats), suggesting heterogeneity in how constraints are routed. The authors conclude that hidden-constraint failure is fundamentally a routing problem rather than a knowledge problem.

The study evaluates prompted interventions and finds none achieve the repair effect of activation patching. All prompted methods instead inflate conservative bias through a single mediation pathway—prerequisite mention—highlighting the limitations of prompt engineering for this class of failures.

The paper contributes a diagnostic framework and empirical evidence that shifts the focus from whether models know constraints to whether they can route them effectively into decisions.