Researchers develop interpretable models to decode why AI safety annotators disagree

A team of researchers has developed an interpretable machine learning approach to understand why annotators assign different safety labels to the same AI outputs. The method, called Annotator Policy Models, learns from annotators' labeling patterns alone, without requiring them to explain their reasoning, a process that would be expensive and often inaccurate.

The researchers identify three distinct sources of annotation disagreement. Operational failures occur when annotators misunderstand the task itself. Policy ambiguity arises when safety instructions are worded in ways that reasonable people interpret differently. Value pluralism reflects genuine disagreement about safety priorities, sometimes patterned along demographic lines. Distinguishing these sources matters: operational failures need better training, ambiguity needs clearer wording, and pluralism needs inclusive deliberation.

In validation experiments, APMs achieved over 80% accuracy in modeling individual annotator policies and correctly predicted how annotators would respond to counterfactual edits to safety scenarios. When applied to real data, the method surfaced both systematic misinterpretations of existing policies and measurable differences in safety priorities across demographic groups.

The work appears on arXiv and has been accepted to ACM FAccT 2026, a peer-reviewed venue focused on fairness, accountability, and transparency in AI systems. The paper includes 38 pages, 13 figures, and seven authors from multiple institutions.