Preprint finds detectable occupational bias in language models despite passing behavioral tests

A new arXiv preprint proposes a causal framework to measure representational bias in language models (LMs), arguing that behavioral tests alone may not capture underlying associations that could still influence outputs. The authors, Keren Fuentes and Aaron Mueller, note that LMs often pass behavioral bias evaluations, but it remains unclear whether this indicates a true absence of bias or merely learned suppression of biased expressions.

The framework decomposes occupational bias into two stages: a model's internal representation of a user's competence and its observable outputs. To operationalize this, the researchers derive "steering vectors" for representations of user expertise and verify that these vectors causally mediate model behavior in both a question-answering task and a hiring task.

Applying the framework to several open-weight models, the study finds that demographic attributes such as gender, race, and socioeconomic status influence a model's internal representation of user expertise. Crucially, these representational biases were detectable even in cases where behavioral metrics showed no disparity between demographics.

The authors demonstrate that these internal biases can influence downstream behavior under intervention, suggesting potential failure modes that behavioral metrics alone may fail to detect. This indicates that even models passing standard bias evaluations may still harbor latent representational biases with real-world consequences.