Study finds language models partially align with typological patterns but diverge in complement systems and action nominal constructions

A new arXiv preprint investigates how grammatical configurations influence language models' acquisition of action nominal constructions (ANCs), which express events in nominal form. The study compares model learning preferences with typological distributions across natural languages.

Researchers constructed 96 synthetic languages that vary in clause word order, noun-phrase word order, alignment, complement system, and ANC strategy. Using a grammar-engineered method, they combined shared semantic input from natural corpora with computational grammars to generate the synthetic data.

For each synthetic language, the team trained GPT-2-small models from scratch and evaluated them using targeted minimal pairs covering 34 grammatical phenomena. The results reveal typological parallels in basic clause configurations but notable divergences in complement systems and ANC strategies.

The study also found that models learned combinations of grammatical features that are rare or unattested in natural languages. The authors argue this partial alignment suggests natural language distributions may reflect both general learning pressures and functional demands not fully captured by current model learning experiments.