FLOPs vs Real Work: The Importance of Replication in AI Efficiency Assessment

A study investigated the relationship between Floating Point Operations (FLOPs) and execution time in AI models, finding that raw FLOPs are an insufficient metric due to varying parallelization efficiencies. While validating the original study's empirical findings, the replication revealed that the proposed $\alpha$-FLOPs estimation formula underestimates execution time on newer hardware, which exhibits instabilities and discontinuities. The research emphasizes the critical need for comprehensive replication packages in hardware-dependent efficiency assessments.