Paradoxical noise preference in RNNs
Preprint
Abstract
In recurrent neural networks (RNNs) used to model biological neural networks, noise is typically introduced during training to emulate biological variability and regularize learning. The expectation is that removing the noise at test time should preserve or improve performance. Contrary to this intuition, we find that continuous-time recurrent neural networks (CTRNNs) often perform best at a nonzero noise level, specifically, the same level used during training. This noise preference typically arises when noise is injected inside the neural activation function; networks trained with noise injected outside the activation function perform best with zero noise. Through analyses of simple function approximation, maze navigation, and single neuron regulator tasks, we show that the phenomenon stems from noise-induced shifts of fixed points (stationary distributions) in the underlying stochastic dynamics of the RNNs. These fixed point shifts are noise-level dependent and bias the network outputs when the noise is removed, degrading performance. Analytical and numerical results show that the bias arises when neural states operate near activation function nonlinearities, where noise is asymmetrically attenuated, and that performance optimization incentivizes operation near these nonlinearities. Thus, networks can overfit to the stochastic training environment itself rather than just to the input-output data. The phenomenon is distinct from stochastic resonance, wherein nonzero noise enhances signal processing. Our findings reveal that training noise can become an integral part of the computation learned by recurrent networks, with implications for understanding neural population dynamics and for the design of robust artificial RNNs.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Flexible multitask computation in recurrent networks utilizes shared dynamical motifsNature Neuroscience, 2024
- Attractor and integrator networks in the brainNature Reviews Neuroscience, 2022
- Phase Coherence Induced by Additive Gaussian and Non-gaussian Noise in Excitable Networks With Application to Burst Suppression-Like Brain SignalsFrontiers in Applied Mathematics and Statistics, 2020
- Effects of Multiplicative Power Law Neural Noise in Visual Information ProcessingNeural Computation, 2011
- Stochastic Resonance in Recurrent Neural Network with Hopfield-Type MemoryNeural Processing Letters, 2009
- What Is Stochastic Resonance? Definitions, Misconceptions, Debates, and Its Relevance to BiologyPLOS Computational Biology, 2009
- Sources of Signal-Dependent Noise During Isometric Force ProductionJournal of Neurophysiology, 2002
- Coherence Resonance in a Noise-Driven Excitable SystemPhysical Review Letters, 1997
- Training with Noise is Equivalent to Tikhonov RegularizationNeural Computation, 1995
- The statistical reliability of signals in single neurons in cat and monkey visual cortexVision Research, 1983