Researchers propose Qistas, a pre-registered protocol for fairness evaluation of LLMs designed for Iraq and portable to other societies
The protocol introduces paired counterfactual testing, power analysis, and structural audits to address cultural specificity and hypothesis pre-registration in LLM fairness benchmarks.
1 source · cross-referenced
- Qistas is a protocol for evaluating LLM fairness tailored to Iraq’s social, confessional, and regional constituencies, designed to be portable to other societies.
- The protocol uses paired counterfactual pairs, differential refusal analysis, and stereotype reliance metrics, analyzed with paired statistical tests and power analysis.
- Authors release authoring tools (taxonomy, checklists, rulings log, audit, and review notebook) and report a semantic review of a 58-item seed bank where 24 of 25 template groups failed at least one check.
- The paper is a protocol specification with no model evaluations; a pre-registration commits hypotheses, metrics, exclusion rules, and analysis plan for the pilot study.
Researchers from Iraq and the U.S. propose Qistas, a protocol designed to evaluate how large language models (LLMs) treat the social, confessional, and regional constituencies of a society, with an initial instantiation for Iraq. The protocol is explicitly framed as portable to other countries, addressing critiques that existing fairness benchmarks are overwhelmingly English-born and American in their social categories.
The protocol measures procedural symmetry across matched counterfactual pairs, differential refusal rates, stereotype reliance under ambiguity, and terminological framing of documented events. All measurements are paired by design and analyzed using paired statistical machinery, including sign-flip permutation tests, exact McNemar tests, template-clustered bootstrap intervals, and Holm-corrected family-wise error control. A power analysis is included to state, before data collection, the smallest detectable effect for each axis after accounting for template clustering design effects.
The authors release a suite of authoring tools alongside the protocol: a pruned 3x5 taxonomy with four cells closed by design, per-type checklists, a cumulative rulings log, a structural audit, and a human review notebook. These tools are intended to standardize the creation and auditing of fairness evaluation items.
In a semantic review of a 58-item seed bank, 24 of 25 template groups failed at least one check, revealing defect patterns such as evaluation criteria leaking into prompts, token-level swaps that break Arabic agreement, epistemically unbalanced pairs, single-theme refusal probes, and unwritten dialect twins counted as coverage. The authors argue these defects are general hazards of counterfactual fairness benchmarks and not specific to the Iraqi context.
The paper explicitly states it is a protocol specification, not a results paper: no LLM has been evaluated using Qistas. The authors have pre-registered the hypotheses, metrics, exclusion rules, and analysis plan that the pilot study will follow, linking to a dedicated registration page.
- Nov 11, 2026 · The Verge — AI
Google DeepMind unveils AlphaGenome 2 to predict genetic variant effects on cell types
Trust79 - Nov 11, 2026 · Google DeepMind — Blog
Google DeepMind unveils AlphaGenome 2 to predict DNA variant effects in specific cell types
Trust84 - Nov 11, 2026 · arXiv cs.AI
Researchers demonstrate auditable human-agent teams for autonomous synchrotron X-ray experiments
Trust79