Prompt formatting can swing LLM benchmark scores by more than 30x, study finds

Researchers propose two metrics—Format Sensitivity Index (FSI) and Parseability Sensitivity Index (PSI)—to quantify how prompt formatting alone can alter benchmark outcomes. FSI captures the range of accuracy changes induced by different prompt wrappers, while PSI measures the corresponding variation in answer parseability.

In experiments spanning 140,000 OpenRouter generations across 7 QA tasks, 5 wrapper families, and 4 instruct models ranging from 7B to 72B parameters, the study finds that mean FSI varies by more than 30x across models. This variance is largely explained by compliance failures in parsing structured outputs.

A fixed-effects regression shows that parseability remains a strong predictor of accuracy even after controlling for task, model, and wrapper, underscoring the importance of schema compliance in benchmark design.

The authors argue that reporting accuracy without accounting for wrapper variance and compliance is statistically fragile, and they provide practical recommendations for both benchmarking and structured-output deployments to improve reliability.