Benchmark finds large language models are unreliable novelty judges for research ideas
A new benchmark from Allen Institute for AI and Hugging Face shows small prompt changes can swing novelty evaluations by more than 50 percentage points, even reversing verdicts on AI-generated ideas.
1 source · cross-referenced
- A new benchmark called Old Ideas, Novel Problems evaluates how reliably LLMs judge the novelty of research ideas.
- Six LLMs—GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6—were tested on human-authored and AI-generated research ideas.
- Changing a few words in a prompt shifted a model’s accuracy in identifying the more novel idea by more than 50 percentage points in one experiment.
- Purpose-built novelty evaluation systems performed worse than a simple prompted Claude Opus 4.6 baseline.
- The benchmark includes a dataset of 154 high-novelty and 145 low-novelty human-authored ideas, plus 154 AI-generated ideas paired with human ideas.
Researchers from the Allen Institute for AI (Ai2) and Hugging Face released a benchmark called Old Ideas, Novel Problems to test how reliably large language models (LLMs) can evaluate the novelty of scientific research ideas.
The benchmark includes two evaluation settings: Human-Only, which pairs 154 high-novelty and 145 low-novelty human-authored research ideas, and Human+Generated, which pairs the same 154 high-novelty human ideas with 154 AI-generated ideas assumed to be less novel.
Six LLMs were evaluated as novelty judges: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6. The study varied factors such as prompts, reasoning budgets, inclusion of related papers, formatting, and aggregation methods to test stability of judgments.
The most pronounced instability came from prompt wording. In one experiment, switching from a prompt that referenced reviewers’ novelty assessments to a simpler prompt caused GPT-5.4’s accuracy on the Human+Generated pairs to drop from 93.5% to 40.9%—a swing of more than 50 percentage points—and reversed its verdicts on more than half of the same pairs.
The same prompt change had a smaller but still substantial effect on the Human-Only dataset, shifting Claude Opus 4.6’s accuracy by 21 percentage points. This suggests novelty judgments can appear stable for human-authored ideas but behave unpredictably for AI-generated ones.
Two purpose-built novelty evaluation systems performed worse than a simple prompted Claude Opus 4.6 baseline, indicating that specialized designs do not necessarily improve reliability.
The benchmark’s data and code are publicly available, along with an automated pipeline for building new evaluation sets. The authors argue that more dependable novelty evaluation is needed if AI is to play a larger role in scientific discovery.
- Nov 6, 2026 · OpenAI — News
OpenAI releases BioBasicsBench to evaluate AI agents on wet-lab science tasks
Trust79 - Nov 3, 2026 · Hugging Face
Allen Institute releases StateBench to test long-context state tracking in language models
Trust79 - Nov 3, 2026 · Hugging Face
CWE-bench v2 expands public cybersecurity evaluation suite to 100 tasks across 91 CWEs
Trust79