Non-linear Neurons with Human-like Apical Dendrite Activations
Authors:
- Mariana-Iuliana GeorgescuMGMariana-Iuliana Georgescu
- Radu Tudor IonescuRIRadu Tudor Ionescu
- Nicolae-Catalin RisteaNRNicolae-Catalin Ristea
- Nicu SebeNSNicu Sebe
Preprint
Abstract
In order to classify linearly non-separable data, neurons are typically organized into multi-layer neural networks that are equipped with at least one hidden layer. Inspired by some recent discoveries in neuroscience, we propose a new neuron model along with a novel activation function enabling the learning of non-linear decision boundaries using a single neuron. We show that a standard neuron followed by the novel apical dendrite activation (ADA) can learn the XOR logical function with 100\% accuracy. Furthermore, we conduct experiments on five benchmark data sets from computer vision, signal processing and natural language processing, i.e. MOROCO, UTKFace, CREMA-D, Fashion-MNIST, and Tiny ImageNet, showing that ADA and the leaky ADA functions provide superior results to Rectified Linear Units (ReLU), leaky ReLU, RBF and Swish, for various neural network architectures, e.g. one-hidden-layer or two-hidden-layer multi-layer perceptrons (MLPs) and convolutional neural networks (CNNs) such as LeNet, VGG, ResNet and Character-level CNN. We obtain further performance improvements when we change the standard model of the neuron with our pyramidal neuron with apical dendrite activations (PyNADA). Our code is available at: https://github.com/raduionescu/pynada.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Dendritic action potentials and computation in human layer 2/3 cortical neuronsScience, 2020
- Deep appearance and motion learning for egocentric activity recognitionNeurocomputing, 2018
- Detecting anomalous events in videos by learning deep representations of appearance and motionComputer Vision and Image Understanding, 2017
- Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in NeocortexFrontiers in Neural Circuits, 2016
- CREMA-D: Crowd-Sourced Emotional Multimodal Actors DatasetIEEE Transactions on Affective Computing, 2014
- The Lure of the UnknownNeuron, 2006
- Approximate Statistical Tests for Comparing Supervised Classification Learning AlgorithmsNeural Computation, 1998
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Learning representations by back-propagating errorsNature, 1986
- The perceptron: A probabilistic model for information storage and organization in the brain.Psychological Review, 1958