A Study of Biologically Plausible Neural Network: The Role and Interactions of Brain-Inspired Mechanisms in Continual Learning
Preprint
Abstract
Humans excel at continually acquiring, consolidating, and retaining information from an ever-changing environment, whereas artificial neural networks (ANNs) exhibit catastrophic forgetting. There are considerable differences in the complexity of synapses, the processing of information, and the learning mechanisms in biological neural networks and their artificial counterparts, which may explain the mismatch in performance. We consider a biologically plausible framework that constitutes separate populations of exclusively excitatory and inhibitory neurons that adhere to Dale's principle, and the excitatory pyramidal neurons are augmented with dendritic-like structures for context-dependent processing of stimuli. We then conduct a comprehensive study on the role and interactions of different mechanisms inspired by the brain, including sparse non-overlapping representations, Hebbian learning, synaptic consolidation, and replay of past activations that accompanied the learning event. Our study suggests that the employing of multiple complementary mechanisms in a biologically plausible architecture, similar to the brain, may be effective in enabling continual learning in ANNs.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Two sparsities are better than one: unlocking the performance benefits of sparse–sparse networksNeuromorphic Computing and Engineering, 2022
- Biological underpinnings for lifelong learning machinesNature Machine Intelligence, 2022
- Embracing Change: Continual Learning in Deep Neural NetworksTrends in Cognitive Sciences, 2020
- Continual learning of context-dependent processing in neural networksNature Machine Intelligence, 2019
- GABAergic Interneurons in the Neocortex: From Cellular Properties to CircuitsNeuron, 2016
- What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory UpdatedTrends in Cognitive Sciences, 2016
- Branch-specific dendritic Ca2+ spikes cause persistent synaptic plasticityNature, 2015
- Stably maintained dendritic spines are associated with lifelong memoriesNature, 2009
- Simplified neuron model as a principal component analyzerJournal of Mathematical Biology, 1982
- Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in positionBiological Cybernetics, 1980