Qwen3.8 27B model shows strong gains with reasoning on word-form addition task

Simon Willison ran a controlled experiment evaluating the Qwen3.8 27B model's ability to add positive integers and return the exact result in English words. The benchmark used 5,070 cases with operand sizes ranging from one to thirteen digits.

In a reasoning-disabled setting, Qwen3.8 27B achieved 23.57% numeric accuracy overall, with performance varying by operand size. Accuracy was 97.04% for one- to three-digit operands but dropped to 6.44% for ten- to thirteen-digit operands. Despite the low numeric accuracy, format compliance remained high at 96.17%, indicating the model often produced word-formatted strings even when incorrect numerically.

When reasoning was enabled, the model's performance improved dramatically. On a 169-case subset, it achieved 98.8% accuracy, failing only two cases. Willison noted that the longer per-pair runtime with reasoning made broader sampling impractical, so the subset prioritized easier cases first.

The reasoning traces included step-by-step manual addition for large numbers, demonstrating the model's ability to decompose arithmetic operations when explicitly prompted. For example, one trace showed alignment and positional addition for operands exceeding 4 trillion, with explicit carry handling.