Researchers propose physics-inspired method to prune large language models by removing transformer blocks

A new paper from researchers at Multiverse Computing and collaborators proposes reformulating the problem of removing transformer blocks from large language models (LLMs) as an Ising optimization problem, enabling more efficient and higher-quality pruning than existing heuristics.

The approach treats each block as a binary variable—keep or remove—mapping the selection task to a constrained binary optimization (CBO) problem equivalent to finding low-energy states of an Ising glass. The energy of the resulting spin system serves as a strong proxy for downstream benchmark performance, allowing the authors to rank candidate configurations without running full evaluations.

The method computes a Hessian matrix from forward and backward passes on a small calibration dataset, capturing both individual block importance and pairwise interactions. This one-time computation enables cheap energy evaluations for any candidate configuration, and the same Hessian can be reused across different compression targets.

For tractable cases, the authors brute-force search billions of configurations on a single GPU; for larger cases, they leverage classical and quantum-inspired solvers (e.g., tabu search, quantum annealing, QAOA) to efficiently find low-energy states. The authors note that finding good low-energy states, rather than the absolute ground state, is sufficient for practical pruning quality.

Empirically, the method achieves substantial gains over prior block-removal approaches. At 50% compression on Llama-3.3-70B-Instruct, the approach improves MMLU performance by nearly 23 percentage points over the best competing block-removal baseline.

The paper argues that existing block-removal methods often rely on mean-field heuristics that ignore interactions between blocks, limiting quality especially at high compression rates. By explicitly modeling these interactions via the Ising formulation, the method generalizes beyond dense transformers and can be applied to heterogeneous architectures.