Decision models compared for automated evals: performance, cost, and threshold tuning

Decision models have recently proliferated as a fast, low-cost alternative to LLM judges for automated evaluation. In a new post, Hamel Husain and Antaripa Saha compare six decision models—Jev 1.13, Perplexity Decider 1.1 27B, Liquid d1 and d1-ultra, Cloudflare Clef Flash, and OpenAI Decisions (GPT-6 Luna)—on a binary failure-detection task using 100 production traces from an apartment-leasing agent. Each model returned a failure probability for every trace, enabling direct comparison of performance, cost, and calibration.

The authors emphasize that using the default decision threshold of 0.50 without tuning can leave substantial performance on the table. They demonstrate how to tune thresholds to balance precision and recall based on task-specific requirements, noting that the optimal tradeoff depends on business goals. For example, automated alerting systems may demand at least 80% precision and recall, which only Perplexity Decider, Liquid d1-ultra, and OpenAI Decisions met after tuning.

Performance differences were pronounced: OpenAI Decisions achieved the highest F1 (0.87) after threshold tuning, while Perplexity Decider was the cheapest at 0.82 cents per 100 eval runs with F1 0.79. The authors present a cost–performance Pareto frontier to help practitioners choose the right model for their budget and quality needs.

Calibration of probability outputs also varied widely. Some models, like Perplexity Decider, Liquid d1, and Clef Flash, moved closer to the diagonal after Platt scaling, while OpenAI’s probabilities degraded slightly. The authors provide prompts and methods to test and improve calibration using stratified cross-validation and Brier scores.

The post concludes with practical guidance: set minimum precision and recall requirements, tune thresholds deliberately, validate calibration, and iterate as prompts and data evolve. The authors note that model performance is highly task-dependent and encourage practitioners to measure their own datasets rather than rely on generic claims.