Researchers propose C-VCE, a diffusion-based framework for visual counterfactual explanations in vision models

Visual counterfactual explanations aim to answer: what minimal change to an image would flip a model’s prediction? Existing diffusion-based methods can produce realistic edits but depend on external classifiers that must work reliably on noisy images, which limits robustness and deployability in safety-critical settings such as medicine.

Researchers introduce C-VCE, a diffusion framework that integrates a classifier directly into the generative model using a concept bottleneck layer. This design guides counterfactual edits with human-interpretable features (concepts) rather than pixel-level edits guided by a separate noise-robust classifier.

C-VCE allows users to toggle semantic concepts during sampling, minimally adjusting relevant image regions while preserving the rest of the image and respecting feature correlations. A probabilistic regularizer balances the trade-off between changing the prediction and staying close to the original, and a gradient-based mask confines modifications to the most relevant regions.

On benchmarks such as CelebA, C-VCE matches or improves flip rates while producing counterfactuals that are visually closer to the input and less distorted than baselines that rely on separate noisy-image classifiers. The authors argue this makes C-VCE a practical tool for vision systems requiring concrete “what-if” images without relying on an additional noise-robust classifier.

The work suggests that exposing and controlling an internal concept layer is a promising direction for making powerful generative models more understandable and safer to use.