NIST launches Artificial Intelligence Technology Evaluation program to standardize AI model testing

The U.S. National Institute of Standards and Technology (NIST) announced the launch of its Artificial Intelligence Technology Evaluation (AITE) program, a new initiative designed to provide researchers with a sequestered testbed environment for evaluating AI model performance. The program aims to address risks of data contamination by using blind data and sequestered environments, ensuring that evaluation datasets are not inadvertently used for training models.

AITE’s initial focus is on image analysis tasks using large vision language models (VLMs), with three domains specified: quantum science, genomics, and public safety. The program will expand the number and variety of tasks over time, according to the announcement.

Participants can engage in two tracks: data providers submit original datasets and associated tasks that remain inaccessible to others, while model providers submit AI models for testing on these datasets. NIST will provide common data, metrics, and scoring infrastructure to facilitate consistent and comparable evaluations.

Model providers will receive detailed measurements of their models’ performance on the datasets and tasks, including comparisons against other models using the same metrics. This structure is intended to improve comparability across models and reduce variability in evaluation outcomes.

Participation is open to all interested parties who agree to the AITE Participation Agreement and rules. NIST has provided contact information for inquiries and specified that additional resources, including evaluation overviews and task specifications, are available for those interested in learning more.