Physics Models LLM Pruning: Mapping Block Removal to Ising Optimization

Article Summary

A new paper details a breakthrough in LLM compression, reframing the challenge of deciding which transformer blocks to prune as a constrained binary optimization (CBO) problem. This problem is mathematically equivalent to finding low-energy states in an Ising glass, a complex disordered spin system. Unlike existing heuristic methods that treat blocks independently, this approach models the critical, interacting couplings between blocks, which is crucial for maintaining model coherence. By reformulating the block selection using the Hessian matrix, the authors establish the system's energy as a strong proxy for the resulting model's performance on benchmarks. Furthermore, they demonstrate the practicality of the method by showing that computing the energy is cheap, and solving the optimization problem can be done efficiently using specialized classical and quantum-inspired solvers, even for massive configurations.

Key Points

• The core innovation is mapping block removal to an Ising glass energy minimization problem, accurately capturing the non-independent, many-body interactions between transformer blocks.
• The calculated 'energy' serves as a reliable, cheap proxy for the actual benchmark performance, bypassing the need for costly model retraining and full benchmarking of candidate configurations.
• The method offers superior performance, yielding significant quality improvements (e.g., 23 percentage points on MMLU at 50% compression of Llama-3.3-70B-Instruct) compared to state-of-the-art pruning techniques.