New expert-validated STEM QA dataset shows frontier models underperform below 25%

Researchers from multiple institutions introduced 'Expert-validated STEM QA,' a dataset designed to address shortcomings in existing STEM evaluation datasets. The dataset consists of 398 questions spanning Physics, Chemistry, Biology, and Mathematics, created by 241 domain experts. The authors identified gaps in prior datasets, including saturated model performance, skewed taxonomy distributions, misalignment with real-world scientific use, and inaccuracies stemming from contest-based data collection and time-bound reviews.

To mitigate these issues, the team implemented a multi-stage validation process: they designed a balanced taxonomy, vetted contributors using quality-driven incentives, conducted multiple rounds of expert reviews, and ensured verifiable question-and-answer formats. The resulting dataset provides a more rigorous benchmark for evaluating STEM reasoning in AI models.

Benchmarking results showed frontier models underperformed on the dataset, scoring below 25%. This suggests significant headroom for improvement in STEM reasoning capabilities among leading AI systems. The authors also demonstrated practical utility by post-training an open-source model on a private subset of 2,000 examples, which improved performance by 15% relative to the baseline on the STEM subset of the HLE-verified dataset (p=0.045).

The dataset has been partially open-sourced for the research community, with additional materials and links to associated code and demos available via arXiv and affiliated platforms.