OpenAI flags reliability issues in SWE-Bench Pro coding benchmark
New analysis from OpenAI questions the accuracy and consistency of a widely used AI coding evaluation suite.
Trust79
HypeLow hype
1 source · single source
- OpenAI published an analysis identifying reliability and accuracy concerns in SWE-Bench Pro, a popular benchmark for evaluating AI coding performance.
- The findings suggest current evaluations may not reliably reflect model capabilities in real-world coding tasks.
OpenAI published an analysis raising concerns about the reliability and accuracy of SWE-Bench Pro, a benchmark commonly used to evaluate AI models on coding tasks.
The analysis suggests that SWE-Bench Pro may not consistently measure what it intends to, calling into question its suitability as a standard for evaluating coding performance in AI systems.
The findings imply that current evaluations could misrepresent model capabilities, making it harder to compare different AI systems fairly or to track progress over time.
- Oct 5, 2026 · arXiv cs.CL
Turkish decision-making benchmark HakemBench released with 2,346 items across seven tracks
Trust79 - Oct 5, 2026 · arXiv cs.CL
XiangqiBench shows frontier LLMs fail in closed-loop agent evaluations despite strong static metrics
Trust79 - Oct 5, 2026 · arXiv cs.CL
Paper proposes Calibrated User Embeddings to improve multi-turn AI agent benchmarking
Trust79