Comparison finds automated evals correlate with human annotations in 100 traces

Applied AI engineer Hamel Husain compared 100 human-annotated traces against automated evaluation systems to assess how well automated methods align with human judgments. The comparison is presented in a blog post titled 'Do Automated Evals Work?' and focuses on the practical reliability of automated evals in applied settings. Husain, who has worked at Airbnb and GitHub and contributed to early LLM research used by OpenAI for code understanding, frames the analysis as part of ongoing work to bring data science rigor to AI evaluation practices.

The analysis centers on a qualitative and quantitative comparison between human annotations and automated eval outputs across the 100 traces. While the post does not disclose the specific automated eval systems used, it emphasizes that the study was designed to measure alignment rather than absolute correctness. The scope is limited to the examined traces and does not claim generalizability to broader or unseen datasets.

Husain’s work is situated within a broader effort to professionalize AI evaluation, including teaching courses on AI evals for engineers and product managers with over 4,500 students from 500 companies, including employees from OpenAI, Anthropic, and Google. The blog post reflects his focus on practical, data-driven approaches to measuring AI system performance, contrasting with anecdotal or purely qualitative evaluation practices.

The post also situates the analysis within a critique of opaque or unspecified evaluation practices in AI product development, arguing that 'It’s Hard to Eval' can indicate a product or process smell. By providing a concrete comparison between human and automated evals, the work contributes to ongoing discussions about standardization and transparency in AI evaluation.