Paper proposes model-consensus framework to rank LLM responses without fixed ground truth

A new paper on arXiv proposes a consensus-based framework for evaluating large language models by measuring relative preference among responses rather than absolute correctness. The approach replaces fixed ground truth with a voting process where a panel of diverse LLMs ranks anonymized candidate responses to identical prompts.

The study evaluates five state-of-the-art LLMs across five domains: programming, general knowledge, safety, logical reasoning, and mathematics. Each model generates responses and independently ranks peer outputs through a structured voting process.

Scores are aggregated into a Relative Intelligence Index (RII), which quantifies how frequently a model’s responses are preferred by other models. The authors emphasize that the RII reflects inter-model preference alignment rather than objective correctness or human judgment.

The framework is positioned as a scalable, model-driven alternative to traditional benchmarks, particularly in scenarios where multiple valid answers exist and correctness alone cannot distinguish nuanced differences in clarity, completeness, and usefulness.

While not directly aligned with human evaluation, the authors cite prior work suggesting that aggregated model preferences can partially correlate with human judgments, motivating the use of model consensus as a proxy signal.