HALO method improves frozen language models with adaptive refinement, outperforming fixed baselines on MMLU-Pro and GPQA-Diamond

Researchers propose HALO, a hybrid adaptive latent-refinement method designed to improve frozen pretrained language models with a small amount of adaptive extra computation. Unlike fixed refinement strategies that either underperform or waste compute, HALO combines a coarse refinement stage with selective second-stage latent refinement applied only to a subset of tokens chosen via token scoring and monotonic token halting.

On a combined benchmark derived from MMLU-Pro and GPQA-Diamond, HALO achieves the best overall average among paper-facing methods, surpassing the frozen backbone and both fixed-1 and fixed-2 baselines. Internal analysis indicates HALO reaches nearly the same token-accuracy level as fixed-2 while using fewer average applied refine steps than fixed-1 and far fewer than fixed-2.

The authors argue that the advantage of HALO stems not from simply adding more refinement steps, but from better allocating refinement effort across tokens. This yields the strongest paper-facing result while reducing measured controller compute compared to either fixed baseline.