Parallelizing Neural Network Models Effectively on GPU by Implementing Reductions Atomically
Open Access
- 8 October 2022
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 451-466
Abstract
Due to the missing of a good orchestration of loop transformations, existing optimizing compilers for deploying neural networks on GPU either parallelize reductions ineffectively or miss the fusion opportunities with other operators. Neural network models thus exhibit sub-optimal performance on GPU. We present a practical approach called Panamera for the effective parallelization of reductions in neural networks on GPU. Panamera first leverages loop coalescing to flatten the loop dimensions of reductions, converting all reduction operators into canonical forms eligible for the polyhedral model. Next, Panamera uses polyhedral transformations to reduce the data movements caused by unfused reductions and perform multi-block hardware binding not considered by many compilers. Finally, Panamera embeds a highly optimized routine implemented using GPU atomic instructions, further improving the performance of neural network models while guaranteeing the correctness of parallel reductions. The experimental results demonstrate the effectiveness of our approach: for single operators our code obtains a mean speedup of 33.7×, 3.5×, 5.4× and 9.6× over cuDNN, CUB, TVM and Ansor, for sub-graphs our approach outperforms cuDNN, TVM and Ansor by 9.5×, 2.6× and 2.7×, and for end-to-end workloads, a tensor compiler integrated with our approach outperforms them by 122.5%, 19.3% and 15.2%.Keywords
Funding Information
- National Natural Science Foundation of China (U20A20226)
This publication has 40 references indexed in Scilit:
- Polyhedral AST Generation Is More Than Scanning PolyhedraACM Transactions on Programming Languages and Systems, 2015
- Polyhedral parallel code generation for CUDAACM Transactions on Architecture and Code Optimization, 2013
- AlphaZ: A System for Design Space Exploration in the Polyhedral Model2013
- Scan detection and parallelization in "inherently sequential" nested loop programs2012
- Thrust2012
- Polyhedron Model2011
- isl: An Integer Set Library for the Polyhedral Model2010
- Improving data locality with loop transformationsACM Transactions on Programming Languages and Systems, 1996
- Some efficient solutions to the affine scheduling problem. I. One-dimensional timeInternational Journal of Parallel Programming, 1992
- Automatic translation of FORTRAN programs to vector formACM Transactions on Programming Languages and Systems, 1987