Researchers propose audit framework for rater state bias in RLHF preference data

A new arXiv pre-print identifies a structured confound in Reinforcement Learning from Human Feedback (RLHF) that arises when annotators' internal states—such as sustained stress or distress—shift their pairwise preference labels over time. The authors argue that these shifts are not merely random noise but state-dependent patterns that can be shared across annotators working under similar conditions. When such patterns propagate through reward modeling and policy optimization, they risk embedding correlated rater state bias into the learned reward signal.

The paper introduces formal definitions for 'rater state shift,' 'rater state confound,' and 'correlated rater state bias,' and proposes 'survival level emotional authenticity' as a measurable response pattern detectable via lexical, pragmatic, discourse, and safety-related features. The authors analyze how correlated rater state bias can survive aggregation and enter learned reward signals, potentially affecting downstream model behavior.

To enable empirical testing, the authors derive five falsifiable predictions and specify effect size thresholds for an initial audit. They also present an audit protocol and pilot study plan designed to be applied to publicly available instruction-tuned models. Crucially, the work explicitly avoids inferring the training history of any specific deployed model, focusing instead on isolating a plausible and testable source of structured bias in RLHF preference data.

The submission history indicates the paper was first posted on April 14, 2026, under the arXiv cs.AI category. While the authors acknowledge funding support, no specific funding amounts or institutional affiliations are listed in the visible metadata.