Sharp Attention Maps Don't Predict Vision-Language Model Reliability, Mechanistic Study Finds

A widely held intuition suggests that vision-language models deliver more trustworthy outputs when their attention mechanisms concentrate sharply on the relevant regions of an image. Researchers from multiple institutions systematically tested this assumption by instrumenting three open-weight VLM families (LLaVA-1.5, PaliGemma, and Qwen2-VL, each 3–7 billion parameters) with a unified mechanistic probing framework called the VLM Reliability Probe (VRP), comparing attention patterns, generation dynamics, and hidden-state geometry against correctness labels across 3,090 samples.

Attention structure showed virtually no correlation with model accuracy, with Pearson correlation coefficients near zero (R_pb=0.001, 95% confidence interval [-0.034, 0.036]). This held even though attention remained causally important for feature extraction; masking the most-attended 30% of image patches reduced accuracy by 8.2 to 11.3 percentage points, statistically significant at p<0.001. The gap between causal necessity and predictive power suggests attention operates differently than the common intuition suggests.

Reliability signals became legible only in deeper, later stages of computation. A simple linear probe trained on single-layer hidden states achieved area-under-the-receiver-operating-characteristic (AUROC) scores exceeding 0.95 on the POPE (Polling-based Object Probing Evaluation) benchmark for two of the three model families. Self-consistency—generating the same answer across multiple independent inference runs—was the strongest behavioral predictor of correctness observed (R_pb=0.43), though it demands ten times the inference cost.

Fine-grained causal ablations at the neuron level uncovered a critical architectural split with implications for model monitoring. Late-fusion architectures like LLaVA concentrated reliability signals in a fragile late-stage bottleneck; removing just the top five neurons identified by the reliability probe reduced object-identification accuracy by 8.3 percentage points. By contrast, early-fusion designs (PaliGemma and Qwen2-VL) distributed reliability widely across layers and tolerated destruction of roughly half their peak hidden-dimension with degradation of one percentage point or less, indicating more distributed, resilient encoding of correctness signals.