Frontier LLMs fail 42% of oncology decision points in new benchmark, study finds

Researchers introduced the Oncology Decision Boundary Benchmark (ODBB), a 2,005-item evaluation spanning NCCN guideline-conformant decisions and colorectal cancer case-specific judgments. The benchmark was designed to move beyond factual recall tests and assess models’ ability to select guideline pathways, escalate care appropriately, and commit to decisions under uncertainty—core competencies for real-world oncology practice.

Nine frontier LLMs—four closed-source and five open-weight families—released between June 2025 and April 2026 were evaluated. A fully deterministic scorer, validated by two oncologists with high inter-rater agreement (Cohen’s weighted κ = 0.939 and 0.790 on a 225-item stratified sample), classified outputs into 14 failure types without relying on additional LLM inference.

When treating the nine models as a pooled super-model, 42.1% of all items (Wilson 95% CI 40.0–44.3%) were answered correctly by none. This failure rate varied by domain: 35.7% of the 1,586 NCCN items and 66.4% of the 419 colorectal-cancer cases were missed by every model. Failures were concentrated in the ability to choose between guideline pathways before reasoning within any specific pathway—a form of clinical meta-judgment the authors identify as a consistent blind spot.

The authors argue this blind spot is unlikely to be resolved by more training data, suggesting it reflects a structural limitation in current architectures. They also found that models optimized for decisiveness (e.g., GPT-5.5 and Gemini 3.1 Pro Preview) made unsafe commitments three to five times more often than more cautious models, without achieving higher overall accuracy. In 3–9% of items, models identified the correct next clinical step but failed to commit to it—failures of decision, not knowledge.

The study concludes that model quality is no longer the primary bottleneck for clinical LLM deployment. Instead, the binding constraint is the assumption that any single model can serve as the sole basis for a clinical decision. Progress, they argue, will require architectures that can detect when a model reaches its competence boundary and automatically route the decision to a clinician.