Hugging Face reports results of community-wide effort to reproduce 2,226 ICML 2026 papers

Hugging Face ran a public hackathon from July 15 to August 2, 2026, inviting participants to reproduce papers from ICML 2026 using coding agents. Over 1,200 community members published 6,816 Trackio logbooks reproducing 2,226 papers—about a third of the conference—within 19 days.

Participants used a range of coding agents (e.g., Claude Code, Codex, Cursor, OpenResearch’s orx) and Hugging Face’s streamlined interface to pull papers, extract core claims, and run experiments. Each run produced a static Hugging Face Space logbook containing the write-up, code, artifacts, and optionally the full agent execution trace as a Hugging Face Dataset.

An automated Logbook Judge (GLM-5.2) issued per-claim verdicts—verified, falsified, toy, or inconclusive—treating each logbook’s self-assessment as untrusted. Participants received compute credits and launched 2,962 cloud jobs; 274 full agent-trace datasets were published.

Aggregating claim-level verdicts, 51% of examined papers (1,103) had at least one claim independently verified. Of those, 266 papers were fully reproduced with every extracted claim verified, and 632 were partially reproduced with nothing falsified. In total, 3,978 individual claims were confirmed with real experiments.

23% of examined papers (496) had at least one claim falsified or contested, including 49 papers where all claims were falsified. Notably, 242 papers saw independent reproduction teams reach opposite verdicts on the same claims, underscoring that reproducibility is adversarial rather than binary.

The remainder fell into intermediate categories: 502 papers had only toy-scale evidence, and 280 had nothing established due to missing artifacts. The organizers noted missing datasets or unreleased checkpoints as common barriers to full reproduction.

The effort surfaced concrete mathematical and experimental flaws in multiple spotlight papers. For example, one participant falsified a claim in “Towards Optimal Robustness in Learning-Augmented Paging,” showing the additive term grows as Θ(log k) rather than O(1), confirmed via re-implementation up to k = 1,024. Another paper, “Attention's forward pass and Frank-Wolfe,” saw counterexamples emerge at t = 224, ~3,800, and 6,416 steps, explaining why prior finite-horizon checks had appeared to verify the claim.