Scalable and Sustainable Deep Learning via Randomized Hashing
Authors:
Open Access
- 4 August 2017
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 445-454
Abstract
Current deep learning architectures are growing larger in order to learn from complex datasets. These architectures require giant matrix multiplication operations to train millions of parameters. Conversely, there is another growing trend to bring deep learning to low-power, embedded devices. The matrix operations, associated with the training and testing of deep networks, are very expensive from a computational and energy standpoint. We present a novel hashing-based technique to drastically reduce the amount of computation needed to train and test neural networks. Our approach combines two recent ideas, Adaptive Dropout and Randomized Hashing for Maximum Inner Product Search (MIPS), to select the nodes with the highest activations efficiently. Our new algorithm for deep learning reduces the overall computational cost of the forward and backward propagation steps by operating on significantly fewer nodes. As a consequence, our algorithm uses only 5% of the total multiplications, while keeping within 1% of the accuracy of the original model on average. A unique property of the proposed hashing-based back-propagation is that the updates are always sparse. Due to the sparse gradient updates, our algorithm is ideally suited for asynchronous, parallel training, leading to near-linear speedup, as the number of cores increases. We demonstrate the scalability and sustainability (energy efficiency) of our proposed algorithm via rigorous experimental evaluations on several datasets.Keywords
All Related Versions
Funding Information
- National Science Foundation (1652131, 1547433)
This publication has 9 references indexed in Scilit:
- Deep Residual Learning for Image Recognition2016
- Query-aware locality-sensitive hashing for approximate nearest neighbor searchProceedings of the VLDB Endowment, 2015
- Mining of Massive Datasets2014
- DSH2014
- Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research GroupsIEEE Signal Processing Magazine, 2012
- An empirical evaluation of deep architectures on problems with many factors of variation2007
- Very sparse random projections2006
- Learning methods for generic object recognition with invariance to pose and lighting2004
- Database-friendly random projections: Johnson-Lindenstrauss with binary coinsJournal of Computer and System Sciences, 2003