Researchers propose LLM-referenced C-test to evaluate potential supraintelligent systems

Researchers have introduced a benchmark designed to evaluate AI systems that may eventually surpass both human and current AI capabilities. The proposed LLM-Referenced C-test is grounded in Levin’s Kt, a computable variant of Kolmogorov complexity, which provides a theoretical upper bound on the difficulty of inverse problems based on the complexity of the underlying mechanism and the time required to run it.

Unlike traditional benchmarks that rely on human-calibrated difficulty or current AI performance, this approach generates test items whose difficulty is measured on a ratio scale in bits. The authors argue that this allows for the creation of fresh, high-difficulty items before any human or frontier model can solve them, avoiding the circularity of saturating benchmarks.

The paper formalizes the measure and outlines a lightweight generation and certification pipeline. It also explores how the power of the reference machine (the LLM used as a representational tool) and the gap between examiner and examinee affect the test outcomes.

In an initial administration of the test to twelve LLMs, the authors report agent characteristic curves that decrease with difficulty, extending into what they term the "supra-intelligence window"—a region of certified high-difficulty items beyond the reach of current frontier models.

The work is presented as a preprint on arXiv and is under review, with code and data made available by the authors. The authors suggest this framework could provide a path to evaluate systems that may one day exceed the capabilities of both humans and the AI systems used to design the tests.