smevals framework released for small-scale model, prompt, and harness evaluation

Simon Willison and Prime Radiant, Jesse Vincent’s applied AI research lab, released smevals, an open-source eval suite designed to run small-scale evaluations of AI models, prompts, and agent harnesses.

The framework introduces a vocabulary for structured evaluation: an eval is a collection of tasks designed to answer a specific question about model capabilities, such as SVG generation. Each task is a concrete challenge, e.g., 'Generate an SVG of a pelican riding a bicycle.'

Evaluations are executed against configs that specify models and optional parameters like system prompts or model settings. A run records the execution of a config on a task, while a runner is the script that performs the run.

Grading is separated from execution: graders apply checks to runs, producing grades that reflect performance against defined criteria. Checks can range from simple string matching to custom scripts or even using other models to evaluate outputs.

The tool supports multiple workflows: running evals across models like gpt-5.5 and claude-opus-4.6, grading results, serving interactive dashboards locally, or building static HTML reports for hosting. Example commands include 'uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6', 'uvx smevals grade path-to-eval/', and 'uvx smevals serve path-to-eval/'.

Willison describes smevals as his third iteration on eval frameworks, emphasizing its focus on simplicity and practicality for small-scale, reproducible testing.