Turkish decision-making benchmark HakemBench released with 2,346 items across seven tracks

HakemBench is a Turkish benchmark designed to evaluate models on typed decision-making tasks, where a model reads a text, a question, and a fixed set of options, then returns a probability for each option. The benchmark’s first release, version 1.0, is fully open under the CC BY 4.0 license and includes 2,346 items and 4,275 questions distributed across seven tracks: fact-check triage, education, guardrails, legal routing, moderation, spam and phishing, and customer support.

A scoring harness accompanies the benchmark to evaluate three metrics: decision quality (macro F1), calibration (via normalized Brier score), and selective automation (via normalized area under the generalized risk-coverage curve). These metrics are combined using a geometric mean, and the harness reports intervals derived from 2,000 bootstrap draws. Additional probes assess sensitivity to option order, paraphrase, English translation, and substituted names.

Gold labels for the benchmark were generated through blind passes of one AI model family, compared against the aggregated votes of a panel of large language models from other model families. The authors note that these labels are not human-verified.

The public leaderboard includes 16 entries, with the top composite score at 0.888. The releasing lab’s own model ranks 7th with a composite score of 0.660; however, the authors flag that this model’s training data was shaped by earlier runs’ test results, particularly affecting its performance in guardrail, moderation, and customer support tracks. When scored only on the other four tracks, the lab’s model achieves a composite score of 0.678, placing it 6th of 16.