New benchmark evaluates LLMs on black-box optimization from natural-language descriptions

Researchers propose Black-Box Optimization Word Problems (BBOWP), a problem setting where a system must infer both a search space and an optimization algorithm from a natural-language description of a black-box optimization task. In this setting, only objective values are observable and the functional form is unavailable, making problem formulation and algorithm selection central challenges.

To support research on BBOWP, the team introduces the BBOWP Benchmark Suite (BBOWP-Bench), a dataset and evaluation framework that pairs each instance with a natural-language problem description, an executable evaluation environment, and a human-designed baseline formulation. This design enables evaluation of both search-space design and algorithm selection.

Using BBOWP-Bench, the authors conduct the first evaluation of large language models (LLMs) on this task. They find that current LLMs can select suitable algorithms based on the given evaluation budget, demonstrating competence in algorithm recommendation under constrained computational resources.

However, the evaluation also reveals limitations: LLMs sometimes struggle with search-space design, particularly in identifying important variables and balancing their ranges when the problem description is less informative or the search space is highly problem-specific. These findings point to areas where automated problem formulation remains challenging.

The BBOWP-Bench dataset and code are publicly available, providing a foundation for further research into automating black-box optimization workflows with LLMs.