Apple proposes GH-ESD to systematically identify grounded error slices in instance-level vision tasks

Apple’s Machine Learning Research team introduced GH-ESD (Grounded Hypothesis-Driven Error Slice Discovery), a generate-and-verify framework designed to identify systematic failures in instance-level vision tasks such as object detection and segmentation.

Existing slice discovery approaches often model failures as clusters in representation space or combinations of predefined attributes, which the authors argue are insufficient for instance-level tasks where failures stem from contextual relational and spatially grounded visual patterns.

GH-ESD reformulates slice discovery as grounded hypothesis generation and statistical verification, constructing relational failure hypotheses using LLM priors and grounded visual evidence.

The framework discovers hypothesis slices at the instance level via Vision Language Models and verifies them through statistical trend analysis over instance-level errors.

To support evaluation, the team introduced GESD (Grounded Error Slice Dataset), a new benchmark providing expert-defined and spatially grounded slices derived from detection and segmentation failures.

In experiments, GH-ESD consistently outperformed baselines on the GESD benchmark, improving Precision@10 by 0.10 (0.73 vs. 0.63) for detection tasks, and also demonstrated support for segmentation scenarios.

The authors report that GH-ESD identifies interpretable slices that facilitate actionable model improvements, offering a pathway to more targeted debugging and robustness enhancements in production systems.