Apple proposes new benchmark to test vision-language models’ ability to infer visual concepts from image sets

Apple’s Machine Learning Research team has introduced Visual Concept Inference from Sets (VICIS), a new task designed to evaluate vision-language models’ (VLMs) ability to infer shared visual concepts from small sets of example images and apply them to new inputs. The task is framed as follows: given a context set of images sharing a common concept and a query image, the model must generate new images that preserve the context-defined concept while remaining consistent with the query.

The research demonstrates that state-of-the-art VLMs perform poorly on VICIS, often ignoring the visual context or defaulting to biased generations. This limitation underscores a broader challenge in multimodal AI: the ability to reason from purely visual information without relying on textual prompts.

To address this gap, Apple proposes a training framework and architecture that learn to infer visual concepts from image sets and extract concept-specific embeddings from queries. The approach aims to enable models to better capture and apply shared visual concepts, improving the accuracy and diversity of generated outputs.

Experiments conducted on synthetic data and large-scale datasets such as ImageNet and WordNet show that the proposed method generates more accurate and diverse outputs compared to existing approaches. Additionally, the model demonstrates improved generalization to unseen concepts and modalities, including sketches, suggesting broader applicability beyond standard photographic images.

The work is authored by Nick Stracke, Kolja Bauer, Josh Susskind, Miguel Angel Bautista Martin, and Björn Ommer, and is presented as part of Apple’s broader research in computer vision and methods and algorithms. The publication is dated July 2026 and is available on Apple’s Machine Learning Research website.