Official conference reviewer guidelines outperform LLM-generated reviewer-imitating guidelines in automated peer review study

A new arXiv preprint examines how different reviewer guideline designs influence the performance of LLM-based automated peer review systems. The study compares official conference reviewer guidelines with LLM-generated reviewer-imitating guidelines derived from high-quality human reviews. According to the authors, official conference guidelines produced review results that were more consistent with human judgments than the LLM-generated alternatives.

The research further finds that enforcing strict rubric-style scoring consistently degraded performance in automated reviews. This suggests that allowing subjective and holistic scoring may be more effective for LLM-based peer review systems. The authors propose that evaluation criteria refined through established conference practices serve as better guidance for automated reviewing than guidelines designed to imitate human reviewers.

The preprint, titled 'Evaluating the Impact of Reviewer Guideline Design on LLM-Based Automated Peer Review,' was submitted to arXiv on May 16, 2026, and includes 18 pages with two figures. The work is positioned within the ACL 2026 Findings track, indicating it is intended as a contribution to the upcoming Association for Computational Linguistics conference.