Token-level entropy fails as confidence signal for small language models under 3B parameters

A new arXiv preprint evaluates seven entropy-based confidence methods for Small Language Models (SLMs) with fewer than 3 billion parameters, finding that token-level entropy is an unreliable signal at this scale. Across 7 model pairs and 5 standard NLU benchmarks, mean token entropy was near zero in 91% of dataset-model combinations, regardless of whether the answer was correct or incorrect. This renders token-based early stopping or uncertainty estimation ineffective for SLMs running on consumer hardware.

The study introduces semantic entropy as a viable alternative, computed by generating multiple samples, clustering answers by meaning, and measuring distributional uncertainty. Using semantic entropy to selectively route uncertain queries to larger expert models yielded accuracy improvements of up to +50 percentage points. The gains varied by routing strategy: cross-family routing (e.g., SmolLM 360M to Phi-3.5-mini) averaged +22.0% improvement, compared to +6.8% for same-family routing, indicating that expert model quality matters more than architectural compatibility.

The authors argue that the value of entropy-based methods in SLMs lies not in computational savings but in intelligent compute allocation—spending more tokens where uncertainty is high. This approach enables targeted escalation to larger models only when necessary, rather than uniformly increasing compute across all queries.

The paper evaluates seven distinct approaches, including token-level entropy early stopping, semantic entropy estimation, and uncertainty-aware routing, across a diverse set of SLMs and benchmarks. The results suggest a paradigm shift from optimizing for efficiency alone to optimizing for accuracy through selective, uncertainty-aware compute allocation.