Why children outperform AI in language learning remains unresolved

Children typically master grammar and begin producing grammatically correct sentences after exposure to roughly 10 million to 30 million words, a scale that remains far below the training data consumed by large language models (LLMs). For context, a modern LLM like Meta’s Llama 3.1 was pretrained on 15 trillion tokens, and frontier models may soon require 10 times that amount.

This disparity is described by researchers as the 'data efficiency gap,' where a preteen in a linguistically rich environment may hear around 100 million words by adolescence, and even with literacy, accumulates at most 300 million words by age 20. By comparison, the training data for a state-of-the-art LLM could equate to the linguistic experience of an entire city across a generation, according to one cognitive scientist.

The gap underscores a longstanding puzzle in cognitive science: how do children achieve linguistic fluency with such limited input? Some researchers, like Michael C. Frank of Stanford, describe the phenomenon as 'totally miraculous,' noting that training a model like GPT-2 on 30 million words yields a 'nonsense generator,' not a child.

The question has theoretical stakes. Noam Chomsky’s argument for innate linguistic knowledge—the 'poverty of the stimulus'—challenged the view that language is learned purely from environmental exposure. While Chomskyan generative grammar dominated linguistics for decades, early AI efforts to encode grammatical rules explicitly largely failed to produce scalable language systems.

Today’s neural approaches, which learn statistical patterns, have succeeded where symbolic methods failed, but they remain data-intensive. Researchers now hope that reverse-engineering child learning could yield AI models that are more data-efficient, potentially benefiting applications like video-based training and supporting minority language communities.

The mystery also extends to enduring questions about human cognition: Is language acquisition driven by innate constraints, or are there universal principles of learning that could apply to both humans and machines? The answers could reshape both AI architecture and our understanding of how children develop language.