Evolving and Understanding Sparse Deep Neural Networks using Cosine Similarity
Authors:
- Joost PieterseJPJoost Pieterse
- Decebal Constantin MocanuDMDecebal Constantin Mocanu
Preprint
Abstract
Training sparse neural networks with adaptive connectivity is an active research topic. Such networks require less storage and have lower computational complexity compared to their dense counterparts. The Sparse Evolutionary Training (SET) procedure uses weights magnitude to evolve efficiently the topology of a sparse network to fit the dataset, while enabling it to have quadratically less parameters than its dense counterpart. To this end, we propose a novel approach that evolves a sparse network topology based on the behavior of neurons in the network. More exactly, the cosine similarities between the activations of any two neurons are used to determine which connections are added to or removed from the network. By integrating our approach within the SET procedure, we propose 5 new algorithms to train sparse neural networks. We argue that our approach has low additional computational complexity and we draw a parallel to Hebbian learning. Experiments are performed on 8 datasets taken from various domains to demonstrate the general applicability of our approach. Even without optimizing hyperparameters for specific datasets, the experiments show that our proposed training algorithms usually outperform SET and state-of-the-art dense neural network techniques. The last but not the least, we show that the evolved connectivity patterns of the input neurons reflect their impact on the classification task.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Regularisation of neural networks by enforcing Lipschitz continuityMachine Learning, 2020
- Multi-Objective Evolutionary Federated LearningIEEE Transactions on Neural Networks and Learning Systems, 2019
- Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network scienceNature Communications, 2018
- A topological insight into restricted Boltzmann machinesMachine Learning, 2016
- Automatic identification of mixed bacterial species fingerprints in a MALDI-TOF mass-spectrumBioinformatics, 2014
- EEG signals classification using the K-means clustering and a multilayer perceptron neural network modelExpert Systems with Applications, 2011
- Indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: Dependence on recording region and brain statePhysical Review E, 2001
- Emergence of Scaling in Random NetworksScience, 1999
- The Organization of Behavior; A Neuropsychological TheoryThe American Journal of Psychology, 1950