AdderNet: Do We Really Need Multiplications in Deep Learning?
Authors:
Top Cited Papers
- 1 June 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1465-1474
Abstract
Compared with cheap addition operation, multiplication operation is of much higher computation complexity. The widely-used convolutions in deep neural networks are exactly cross-correlation to measure the similarity between input feature and convolution filters, which involves massive multiplications between float values. In this paper, we present adder networks (AdderNets) to trade these massive multiplications in deep neural networks, especially convolutional neural networks (CNNs), for much cheaper additions to reduce computation costs. In AdderNets, we take the L1-norm distance between filters and input feature as the output response. The influence of this new similarity measure on the optimization of neural network have been thoroughly analyzed. To achieve a better performance, we develop a special back-propagation approach for AdderNets by investigating the full-precision gradient. We then propose an adaptive learning rate strategy to enhance the training procedure of AdderNets according to the magnitude of each neuron's gradient. As a result, the proposed AdderNets can achieve 74.9% Top-1 accuracy 91.7% Top-5 accuracy using ResNet-50 on the ImageNet dataset without any multiplication in convolutional layer. The codes are publicly available at: (https://github.com/huaweinoah/AdderNet).Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression2017
- Learning from Multiple Teacher Networks2017
- A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning2017
- Deep Learning with Low Precision by Half-Wave Gaussian Quantization2017
- Aggregated Residual Transformations for Deep Neural Networks2017
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Deep Residual Learning for Image Recognition2016
- Fully convolutional networks for semantic segmentation2015
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998