Replication study finds FLOPs-based efficiency metrics unreliable on modern hardware

A new paper on arXiv (arXiv:2608.14550) replicates a widely cited efficiency metric, α-FLOPs, and finds it unreliable when applied to newer, more powerful hardware. The α-FLOPs formula was proposed to estimate computational cost beyond raw FLOPs by accounting for differences in how operations parallelize across hardware. However, the replication study by Enrique Barba Roque and Luís Cruz reveals that execution time on modern accelerators exhibits instabilities and discontinuities — including jumps and oscillations — that the α-FLOPs formula systematically underestimates.

The authors identify limitations in the original study’s replication materials, noting missing dependency details and lack of transparency in regression data. These gaps complicate independent verification and highlight broader challenges in hardware-dependent AI efficiency research. Despite these issues, the replication confirms the core thesis that raw FLOPs alone are insufficient for predicting execution time, as spatial dimensions are more easily parallelized than kernel dimensions.

Fine-grained measurements in the study show that the relationship between FLOPs and execution time is more complex than previously understood. On newer hardware, execution time can vary non-monotonically with FLOPs, defying simple predictive models. The authors argue that this complexity arises from hardware-specific behaviors, including memory hierarchies, kernel fusion, and scheduling policies, which are not captured by FLOPs-based metrics.

To support reproducibility, the authors release a complete replication package, including code and data, aimed at improving transparency in AI efficiency research. They emphasize the need for standardized, hardware-aware evaluation protocols and full disclosure of experimental conditions to enable credible comparisons across studies.