New benchmark shows frontier LLMs discovering novel cryptanalytic attacks

A new benchmark called CryptanalysisBench evaluates the ability of large language models (LLMs) to perform mathematical cryptanalysis, a domain at the intersection of mathematical reasoning and cybersecurity. The benchmark consists of 191 tasks across six families of cryptographic primitives, primarily drawn from four NIST standardization competitions. These tasks are organized into three tiers: (i) primitives with known practical breaks, (ii) primitives with no known practical break evaluated at full strength and as scaled-down variants, and (iii) a challenge set of production primitives at the frontier of cryptanalysis.

Five frontier models—Claude Opus 4.8, Sonnet 5, Mythos 5, GPT-5.5, and the open-weights GLM-5.2—were tested on CryptanalysisBench. The models broke 65%–86% of Tier 1 schemes, 6–12 Tier-2 schemes at full strength, and 24–61 scaled-down variants. Beyond reproducing known results, the models also produced novel cryptanalysis, including a key-recovery attack that exploits a design flaw in the SpoC AEAD and an error in KINDI’s published CCA-security proof, both previously unknown to the authors.

Anthropic used CryptanalysisBench to test its Mythos Preview model, discovering new vulnerabilities in the Hawk authenticated encryption scheme and reduced-round variants of AES. The benchmark is intended to serve as a tool for tracking whether AI cryptanalysis becomes a serious factor in the field and as a scaffold for stress-testing cryptographic schemes before deployment.

The authors emphasize that the attacks surfaced by the benchmark represent an early snapshot of a rapidly evolving frontier. They suggest that AI-driven cryptanalysis may soon match or exceed the published state of the art, highlighting the need for ongoing evaluation and adaptation in cryptographic research and practice.