Efficient-Grad: Efficient Training Deep Convolutional Neural Networks on Edge Devices with Grad ient Optimizations
Open Access
- 8 February 2022
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Embedded Computing Systems
- Vol. 21 (2) , 1-24
Abstract
With the prospering of mobile devices, the distributed learning approach, enabling model training with decentralized data, has attracted great interest from researchers. However, the lack of training capability for edge devices significantly limits the energy efficiency of distributed learning in real life. This article describes Efficient-Grad, an algorithm-hardware co-design approach for training deep convolutional neural networks, which improves both throughput and energy saving during model training, with negligible validation accuracy loss. The key to Efficient-Grad is its exploitation of two observations. Firstly, the sparsity has potential for not only activation and weight, but gradients and the asymmetry residing in the gradients for the conventional back propagation (BP). Secondly, a dedicated hardware architecture for sparsity utilization and efficient data movement can be optimized to support the Efficient-Grad algorithm in a scalable manner. To the best of our knowledge, Efficient-Grad is the first approach that successfully adopts a feedback-alignment (FA)-based gradient optimization scheme for deep convolutional neural network training, which leads to its superiority in terms of energy efficiency. We present case studies to demonstrate that the Efficient-Grad design outperforms the prior arts by 3.72x in terms of energy efficiency.Keywords
Funding Information
- Hong Kong Research Grants Council under General Research Fund (16215620)
- HKUST-Qualcomm Joint Innovation and Research Laboratory
This publication has 40 references indexed in Scilit:
- In-Datacenter Performance Analysis of a Tensor Processing Unit2017
- Deep Convolutional Neural Network Architecture With Reconfigurable Computation PatternsIEEE Transactions on Very Large Scale Integration (VLSI) Systems, 2017
- Random synaptic feedback weights support error backpropagation for deep learningNature Communications, 2016
- Deep Residual Learning for Image Recognition2016
- ShiDianNao2015
- DaDianNao: A Machine-Learning Supercomputer2014
- DianNaoACM SIGPLAN Notices, 2014
- Chisel2012
- CACTI-P: Architecture-level modeling for SRAM-based structures with advanced leakage reduction techniques2011
- Learning representations by back-propagating errorsNature, 1986