Researchers propose theoretical framework for language generation that tolerates controlled hallucinations

Researchers Irene Strauss, Alexandra Butoi, and Ryan Cotterell propose a new theoretical framework for language generation that explicitly tolerates controlled hallucinations. The work, published as an arXiv preprint (arXiv:2606.28354), revisits the "generation in the limit" paradigm, which shifts the objective from language identification to producing valid, unseen strings from a target language.

The authors introduce a new notion of precision and recast the problem as a recall-precision trade-off, analyzing generation under constraints on enumeration, novelty, and validity. A key contribution is the analysis of learners that are not eventually valid: they allow infinitely many mistakes, provided their frequency tends to zero so that precision remains one.

The paper shows that this relaxation can strictly increase recall when the adversary permanently withholds a large portion of the target language. This suggests that allowing rare errors may improve coverage of the target language in practice.

The authors also study a continuous relaxation of the novelty constraint, requiring only a fixed fraction of outputs to be novel. This further aligns the framework with settings encountered by large language models, where occasional repetitions or near-repetitions are common.

Taken together, the results move toward a more realistic model of language generation where occasional errors and repetitions are unavoidable, but their rates are controlled. The framework provides a formal basis for reasoning about trade-offs between coverage, validity, and novelty in language generation systems.