New pruning method outperforms magnitude-based approaches by leveraging Fisher information geometry

A new pruning scheme introduced in a preprint on arXiv derives its selection criterion from the differential-geometric distance in model space, specifically the geodesic distance induced by the Fisher information metric. In this framework, pruning a parameter is treated as moving the model to a hypersurface where that parameter is zero, and the minimal distance to that hypersurface quantifies the true change in model behavior caused by pruning.

The authors argue that traditional magnitude pruning and methods based on local Fisher information alone are suboptimal because they do not account for the global geometry of the model space. By progressively approximating the geodesic distance, they derive a hierarchy of pruning methods that includes magnitude pruning as the simplest case and more sophisticated schemes as better approximations of the geodesic distance.

The proposed method is evaluated on fully-connected networks and vision transformers using the MNIST and CIFAR-10 datasets. Across the full pruning range from 0% to 100% and across five random seeds, the method outperforms both magnitude-based pruning and pruning guided by local Fisher information on accuracy and the Matthews correlation coefficient.

The paper also explores intermediate approximations of the geodesic distance that balance computational efficiency with performance. These approximations yield pruning schemes that are computationally tractable while maintaining near-optimal results, suggesting practical pathways for deployment in resource-constrained settings.