AI models struggle with spatial, memory, and abstract reasoning puzzles designed to expose gaps in capabilities

Puzzles and games have long served as a proving ground for AI capabilities, tracing back to Arthur Samuel’s 1959 checkers-playing algorithm and classic board games like chess and Go. Recent evaluations show that AI models have made measurable progress on certain puzzle benchmarks, with solve rates on New York Times Connections puzzles rising from 18% in late 2024 to near-perfect by early 2025.

However, models continue to falter on tasks where human cognition excels, particularly spatial reasoning. Mental rotation puzzles—common in IQ tests—require identifying whether different images represent the same object from another angle. Despite advances in vision-capable models, these puzzles remain a weak spot, underscoring a gap between language-centric training and spatial manipulation abilities.

Models also struggle when confronted with memory traps: puzzles that closely resemble training data but include subtle variations. In a 2024 study by researchers from Google and the University of Illinois Urbana-Champaign, models trained on classic Knights and Knaves logic puzzles were tested on slightly altered versions. Even top-tier models often failed to adapt, defaulting to memorized responses rather than reasoning through the new constraints. Similar pitfalls appear in SimpleBench-style arithmetic and logic problems that mimic more complex training examples.

Abstract and visual reasoning puzzles, such as those in the ARC-AGI benchmark, further expose limitations. Models may solve these puzzles by learning non-generalizable, overly complex rules rather than leveraging simple visual concepts that humans intuitively apply. While model performance on ARC-AGI has improved over the past year, certain puzzles remain unsolved, revealing persistent gaps in generalization.

Human and AI strengths diverge in other ways. Intuition-based problems often exploit human cognitive biases, causing people to respond reflexively while models deliberate more systematically. Conversely, scale-dependent tasks like the Tower of Hanoi show that models can excel on simple versions but face challenges as complexity increases, highlighting a reliance on pattern matching over robust algorithmic reasoning.