Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware
Open Access
- 6 July 2020
- journal article
- research article
- Published by Springer Nature in Neural Computing and Applications
- Vol. 33 (7) , 2589-2604
Abstract
Artificial neural networks (ANNs) have emerged as hot topics in the research community. Despite the success of ANNs, it is challenging to train and deploy modern ANNs on commodity hardware due to the ever-increasing model size and the unprecedented growth in the data volumes. Particularly for microarray data, the very high dimensionality and the small number of samples make it difficult for machine learning techniques to handle. Furthermore, specialized hardware such as graphics processing unit (GPU) is expensive. Sparse neural networks are the leading approaches to address these challenges. However, off-the-shelf sparsity-inducing techniques either operate from a pretrained model or enforce the sparse structure via binary masks. The training efficiency of sparse neural networks cannot be obtained practically. In this paper, we introduce a technique allowing us to train truly sparse neural networks with fixed parameter count throughout training. Our experimental results demonstrate that our method can be applied directly to handle high-dimensional data, while achieving higher accuracy than the traditional two-phase approaches. Moreover, we have been able to create truly sparse multilayer perceptron models with over one million neurons and to train them on a typical laptop without GPU (https://github.com/dcmocanu/sparse-evolutionary-artificial-neural-networks/tree/master/SET-MLP-Sparse-Python-Data-Structures), this being way beyond what is possible with any state-of-the-art technique.Keywords
All Related Versions
This publication has 42 references indexed in Scilit:
- The WEKA data mining softwareACM SIGKDD Explorations Newsletter, 2009
- Feature selection for high-dimensional dataComputational Management Science, 2008
- MapReduceCommunications of the ACM, 2008
- Distributed computation of the knn graph for large high-dimensional point setsJournal of Parallel and Distributed Computing, 2007
- Feature-based image registration by means of the CHC evolutionary algorithmImage and Vision Computing, 2006
- Microarray Gene Expression Profiling of B-Cell Chronic Lymphocytic Leukemia Subgroups Defined by Genomic Aberrations and VH Mutation StatusJournal of Clinical Oncology, 2004
- Gene Expression Profiling of Gliomas Strongly Predicts SurvivalCancer Research, 2004
- Wrappers for feature subset selectionArtificial Intelligence, 1997
- Floating search methods in feature selectionPattern Recognition Letters, 1994
- Learning representations by back-propagating errorsNature, 1986