ADA-GP: Accelerating DNN Training By Adaptive Gradient Prediction
Open Access
- 28 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1092-1105
Abstract
Neural network training is inherently sequential where the layers finish the forward propagation in succession, followed by the calculation and back-propagation of gradients (based on a loss function) starting from the last layer. The sequential computations significantly slow down neural network training, especially the deeper ones. Prediction has been successfully used in many areas of computer architecture to speed up sequential processing. Therefore, we propose ADA-GP, which uses gradient prediction adaptively to speed up deep neural network (DNN) training while maintaining accuracy. ADA-GP works by incorporating a small neural network to predict gradients for different layers of a DNN model. ADA-GP uses a novel tensor reorganization method to make it feasible to predict a large number of gradients. ADA-GP alternates between DNN training using backpropagated gradients and DNN training using predicted gradients. ADA-GP adaptively adjusts when and for how long gradient prediction is used to strike a balance between accuracy and performance. Last but not least, we provide a detailed hardware extension in a typical DNN accelerator to realize the speed up potential from gradient prediction. Our extensive experiments with fifteen DNN models show that ADA-GP can achieve an average speed up of 1.47 × with similar or even higher accuracy than the baseline models. Moreover, it consumes, on average, 34% less energy due to reduced off-chip memory accesses compared to the baseline accelerator.Keywords
All Related Versions
Funding Information
- NSF (National Science Foundation) (2301334)
This publication has 20 references indexed in Scilit:
- Efficient Processing of Deep Neural Networks: A Tutorial and SurveyProceedings of the IEEE, 2017
- Densely Connected Convolutional Networks2017
- Using Dataflow to Optimize Energy Efficiency of Deep Neural Network AcceleratorsIEEE Micro, 2017
- Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks2016
- Deep Residual Learning for Image Recognition2016
- Rethinking the Inception Architecture for Computer Vision2016
- ShiDianNao2015
- The Pascal Visual Object Classes Challenge: A RetrospectiveInternational Journal of Computer Vision, 2014
- Efficient BackProp2012
- A Massively Parallel Coprocessor for Convolutional Neural Networks2009