Prompt formatting can swing LLM benchmark scores by more than 30x, study finds
Researchers introduce the Format Sensitivity Index (FSI) and Parseability Sensitivity Index (PSI) to quantify how prompt wrappers affect model accuracy and answer parseability across 140,000 generations.
1 source · cross-referenced
- Prompt wrappers that differ only in formatting can alter LLM benchmark scores enough to reverse leaderboard rankings.
- The newly introduced Format Sensitivity Index (FSI) measures the range of accuracy changes induced by wrapper choice, while the Parseability Sensitivity Index (PSI) tracks corresponding shifts in answer parseability.
- Across 140,000 OpenRouter generations spanning 7 QA tasks, 5 wrapper families, and 4 instruct models (7B–72B parameters), mean FSI varies by over 30x across models.
- A fixed-effects regression indicates parseability remains a strong predictor of accuracy even after controlling for task, model, and wrapper.
Researchers propose two metrics—Format Sensitivity Index (FSI) and Parseability Sensitivity Index (PSI)—to quantify how prompt formatting alone can alter benchmark outcomes. FSI captures the range of accuracy changes induced by different prompt wrappers, while PSI measures the corresponding variation in answer parseability.
In experiments spanning 140,000 OpenRouter generations across 7 QA tasks, 5 wrapper families, and 4 instruct models ranging from 7B to 72B parameters, the study finds that mean FSI varies by more than 30x across models. This variance is largely explained by compliance failures in parsing structured outputs.
A fixed-effects regression shows that parseability remains a strong predictor of accuracy even after controlling for task, model, and wrapper, underscoring the importance of schema compliance in benchmark design.
The authors argue that reporting accuracy without accounting for wrapper variance and compliance is statistically fragile, and they provide practical recommendations for both benchmarking and structured-output deployments to improve reliability.
- Aug 26, 2026 · arXiv cs.AI
Multi-agent framework lets LLM agents design and run controlled experiments with simulation models
Trust79 - Aug 25, 2026 · Apple — Machine Learning Research
Apple proposes STARFlow2, a unified multimodal model architecture combining normalizing flows with frozen pretrained vision-language models
Trust79 - Aug 25, 2026 · arXiv cs.CL
Study evaluates hybrid retrieval and LLM-assisted query expansion for Khmer semantic search
Trust79