Cross-model auditing improves bias resistance in LLM judges by selecting auditors per bias type

Large language models are increasingly used as automated judges, but their judgments remain susceptible to cognitive biases such as bandwagon, authority, distraction, and sycophancy. Existing mitigations rely on prompt-driven debiasing, which often fails to generalize across bias types, or human evaluation, which does not scale to large volumes of model outputs.

The authors propose Chain-of-Models (CoM), an automated audit pipeline in which a second model inspects the first model’s reasoning trace before producing the final judgment. The key design question is whether the auditor should be the same model, a same-family model, or a different-family model. Across nine models spanning six families, four cognitive biases, and four factual datasets, the study finds that auditor identity matters in two ways.

First, standalone bias resistance does not predict audit effectiveness: Kimi-K2.5 is the strongest standalone model on several biases, yet performs poorly when auditing Qwen2.5-72B’s biased traces. Second, the best auditor is bias-specific: GPT-4o is strongest on bandwagon, authority, and distraction, while GLM-5 is strongest on sycophancy.

The authors operationalize these findings with a per-bias auditor selection rule that scores candidates along functional diversity, per-bias standalone resistance, and calibrated audit effectiveness. Under a calibration/test split, the selector achieves the highest accuracy across the four biased slices at 0.884, compared to 0.824 for the strongest single fixed auditor and 0.805 for the no-audit baseline.

The paper releases data, configurations, and an LLM-agent skill to support reproducibility and further research.