Review finds Anthropic’s new auto-eval plugin pushes users to create evals before inspecting data

Anthropic’s claude-api plugin now includes `build_eval` and `hill-climb` commands intended to help users design evals, validate graders, and improve applications against those evals. The tool automates parts of the eval design process, but a new review by applied AI engineer Hamel Husain argues it pushes users to create evals before inspecting their data.

Husain and collaborator Isaac Flath tested the plugin on conversation traces from an apartment leasing assistant. They report that the tool recommended creating an eval for call-transfer failures before Husain and Flath had reviewed the raw conversations, making it difficult to judge whether the suggested eval targeted a real or high-priority failure mode.

The review also criticizes the tool’s approach to validating judgments, noting that it initially presented data in Markdown files and asked users to report labeling errors in chat rather than in an annotation interface. Husain argues that eval workflows should prioritize in-situ data exploration and labeling, and that the tool’s early-stage design added friction to this process.

Husain further contends that the evaluator produced by the tool bundled multiple failure modes into a single grader, making it hard to interpret results and refine individual checks. He recommends scoping evals narrowly and separating code-based checks from LLM-as-judge approaches to improve clarity and maintainability.