Brain-like learning with exponentiated gradients
Preprint
Abstract
Computational neuroscience relies on gradient descent (GD) for training artificial neural network (ANN) models of the brain. The advantage of GD is that it is effective at learning difficult tasks. However, it produces ANNs that are a poor phenomenological fit to biology, making them less relevant as models of the brain. Specifically, it violates Dale’s law, by allowing synapses to change from excitatory to inhibitory, and leads to synaptic weights that are not log-normally distributed, contradicting experimental data. Here, starting from first principles of optimisation theory, we present an alternative learning algorithm, exponentiated gradient (EG), that respects Dale’s Law and produces log-normal weights, without losing the power of learning with gradients. We also show that in biologically relevant settings EG outperforms GD, including learning from sparsely relevant signals and dealing with synaptic pruning. Altogether, our results show that EG is a superior learning algorithm for modelling the brain with ANNs.Keywords
This publication has 45 references indexed in Scilit:
- A synaptic learning rule for exploiting nonlinear dendritic computationNeuron, 2021
- Distinct in vivo dynamics of excitatory synapses onto cortical pyramidal neurons and parvalbumin-positive interneuronsCell Reports, 2021
- Nonparametric Statistical Methods2015
- The log-dynamic brain: how skewed distributions affect network operationsNature Reviews Neuroscience, 2014
- Context-dependent computation by recurrent dynamics in prefrontal cortexNature, 2013
- Sleep and waking modulate spine turnover in the adolescent mouse cortexNature Neuroscience, 2011
- The tempotron: a neuron that learns spike timing–based decisionsNature Neuroscience, 2006
- Exponentiated Gradient versus Gradient Descent for Linear PredictorsInformation and Computation, 1997
- Learning Quickly When Irrelevant Attributes Abound: A New Linear-Threshold AlgorithmMachine Learning, 1988
- The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programmingUSSR Computational Mathematics and Mathematical Physics, 1967