Researchers train language models on a K–5 curriculum to study knowledge acquisition boundaries

Researchers trained three language models at 0.6B, 1.3B, and 5B scales from scratch on an 88B-token corpus filtered to the U.S. elementary-school curriculum (K–5), excluding concepts, facts, and vocabulary above Grade 5. The dataset, LittleCurriculum, was distilled from FineWeb-Edu through a five-stage filtering pipeline aligned with Common Core standards.

Each model was paired with a matched unfiltered control sharing the same architecture, token count, and training recipe to enable clean comparisons. The authors released base, GRPO (math-specialized), and chatty variants, along with a live chat demo in the browser.

Experiments showed that scaling, supervised fine-tuning with GRPO, and in-context learning improved performance within the curriculum and modestly extended performance along the same learning trajectory, but none meaningfully improved performance on problems requiring capabilities outside the training exposure.

The authors highlight three research directions enabled by the controlled setting: using reinforcement learning to study capability emergence under a restricted prior, studying continual learning by introducing new concepts near the boundary, and comparing machine and child learners on the same exposure.

The paper and models are available on the project site and arXiv, with a live chat interface to interact with the 5B model.