Zyphra releases ZAYA1-8B, a 700M-parameter reasoning model competitive with larger alternatives

Zyphra has published a technical report on ZAYA1-8B, a mixture-of-experts language model designed specifically for reasoning tasks. The model contains 700M active parameters drawn from a total of 8B parameters, built on Zyphra's MoE++ architecture. The entire training pipeline—pretraining, midtraining, and supervised fine-tuning—was executed on AMD's compute, networking, and software platform, establishing the model as a result of a complete AMD stack implementation.

The authors report that ZAYA1-8B matches or exceeds the performance of DeepSeek-R1-0528 on several challenging mathematics and coding benchmarks, despite operating with substantially fewer active parameters. The model was trained from inception for reasoning, incorporating reasoning-focused data from the pretraining phase onward through an answer-preserving trimming scheme. Post-training employs a four-stage reinforcement learning cascade: initial reasoning warmup on mathematical and puzzle tasks, a 400-task curriculum via RLVE-Gym, combined math and code reinforcement learning using test-time compute traces and synthetic environments derived from competitive programming references, and finally behavioral RL to optimize performance on chat and general instruction-following tasks.

The report introduces Markovian RSA, a novel test-time compute method designed to recursively aggregate multiple parallel reasoning traces while constraining the reasoning state carried forward between rounds to only 4K tokens. Under this test-time compute approach, ZAYA1-8B achieves 91.9% accuracy on AIME'25 and 89.6% on HMMT'25—significant performances that narrow the gap to substantially larger open models including Gemini-2.5 Pro, DeepSeek-V3.2, and GPT-5-High. The technical details of the training process, architectural choices, and evaluation methodology are documented across 18 named authors affiliated with Zyphra.