Researchers propose automated benchmark generation for neural relational reasoning using LLMs

A new arXiv preprint introduces Project Auto-World, a framework that uses large language models (LLMs) to automate the generation of challenging benchmark instances for neural relational reasoning. The work targets a persistent challenge in evaluating neural models: determining what makes a problem instance hard and ensuring models can generalize to instances beyond their training distribution.

The authors frame the problem as a search for sampling functions that produce increasingly difficult problem instances. They employ LLM-driven evolutionary search—based on the FunSearch paradigm—and autonomous agentic search to discover these functions. The approach is applied within worlds defined by Datalog rules, with an Edge Transformer serving as the reasoning evaluator.

The paper demonstrates that the Edge Transformer can be improved using data generated by this process, yielding better generalization to unseen data perturbations. The authors also show that the same machinery can be extended to novel worlds proposed by LLMs, suggesting a path toward autonomous research workflows in neural relational reasoning.

The preprint is submitted to the NeurIPS 2026 Exposition & Demonstrations track and includes a link to an associated code repository. The authors include Anirban Das, Joanne Boisson, Irtaza Khalid, Sumita Garai, and Steven Schockaert.