Nicol Spencer Harper

The retina’s role in visual processing has been viewed as two extremes: an efficient compressor of incoming visual stimuli, akin to a camera, or a predictor of future stimuli. Addressing this dichotomy, we developed a spiking neural network model of the retina trained on natural movies under metabolic-like constraints to either encode the present or to predict future scenes. When optimized for efficient temporal prediction ~100 ms into the future, the model not only captures retina-like receptive fields and their mosaic-like organizations, but also exhibits complex retinal processes such as latency coding, motion anticipation, differential motion tuning, and stimulus-omission responses. Notably, the temporal prediction model also accurately predicts the way retinal ganglion cells respond across different animal species to natural images and movies. Our findings suggest that the retina is not merely a compressor of visual input, but rather is fundamentally organized to provide the brain with foresight into the visual world.

A major goal of neuroscience is to identify general principles that can explain the diverse structures and functions of the brain. The principle of temporal prediction provides one approach, arguing that the sensory brain is optimized to represent stimulus features that efficiently predict the immediate future input. Previous work has demonstrated that feedforward hierarchical temporal prediction models can capture the tuning properties of neurons along the visual pathway, and that recurrent temporal prediction models can explain local functional connectivity within primary visual cortex. However, the visual system is also characterized by extensive inter-areal feedback recurrency, which existing models lack. We aimed to better account for the dynamic features of neurons in the visual cortex by incorporating both local recurrency and inter-areal feedback connectivity into a hierarchical temporal prediction model. The resulting model captured tuning properties along the dorsal visual pathway, including pattern motion selectivity and surround suppression, and the contribution of inter-areal connectivity to these properties. Moreover, compared with several alternative normative models, the hierarchical recurrent temporal prediction model provided the closest fit to these tuning properties and was best able to explain neuronal responses to natural stimuli. Accordingly, temporal prediction accounts well for information processing along the visual pathway.

Memory is essential for the neural processing of natural sounds. It has been proposed that cortical memory is subserved by gated recurrency, a powerful machine learning method that enables memory of sequential dependencies. However, standard forms lack biological realism. We built a computational model using a subtractive form of gated recurrency, consistent with cortical circuitry and feedback that resembles corticofugal projections. This architecture outperformed state-of-the-art models with delay lines or non-biological gated recurrency at predicting auditory cortical responses to natural sounds. Its performance was further improved by incorporating a network layer that mirrors the high neuronal density in superficial cortical layers. Analyzing the model, we found that memory retention is longer in secondary than in primary auditory cortex, and that gated recurrency particularly helps capture responses during abrupt changes and quiet periods in the sounds. This work lays out the fundamental functional circuitry of the auditory pathway for processing the sequential dependencies in natural sounds.

Neurons in primary visual cortex (V1) show a remarkable functional specificity in their pre- and postsynaptic partners. Recent work has revealed a variety of wiring biases describing how the short- and long-range connections of V1 neurons relate to their tuning properties. However, it is less clear whether these connectivity rules are based on some underlying principle of cortical organization. Here, we show that the functional specificity of V1 connections emerges naturally in a recurrent neural network optimized to predict upcoming sensory inputs for natural visual stimuli. This temporal prediction model reproduces the complex relationships between the connectivity of V1 neurons and their orientation and direction preferences, the tendency of highly connected neurons to respond more similarly to natural movies, and differences in the functional connectivity of excitatory and inhibitory V1 populations. Together, these findings provide a principled explanation for the functional and anatomical properties of early sensory cortex.

Neurons in primary visual cortex (V1) respond to natural scenes with a sparse and irregular spike code that is carefully balanced by an interplay between excitatory and inhibitory neurons. These neuron classes differ in their spike statistics, tuning preferences, connectivity statistics and temporal dynamics. To date, no single computational principle has been able to account for these properties. We developed a recurrently connected spiking network of excitatory and inhibitory units trained for efficient temporal prediction of natural movie clips. We found that the model exhibited simple and complex cell-like tuning, V1-like spike statistics, and, notably, also captured key differences between excitatory and inhibitory V1 neurons. This suggests that these properties collectively serve to facilitate efficient prediction of the sensory future.

The retina’s role in visual processing has been viewed as two extremes: an efficient compressor of incoming visual stimuli akin to a camera, or as a predictor of future stimuli. Addressing this dichotomy, we developed a biologically-detailed spiking retinal model trained on natural movies under metabolic-like constraints to either encode the present or to predict future scenes. Our findings reveal that when optimized for efficient prediction ∼ 100 ms into the future, the model not only captures retina-like receptive fields and their mosaic-like organizations, but also exhibits complex retinal processes such as latency coding, motion anticipation, differential tuning, and stimulus-omission responses. Notably, the predictive model also more accurately predicts the way retinal ganglion cells respond across different animal species to natural images and movies. Our findings demonstrate that the retina is not merely a compressor of visual input, but rather is fundamentally organized to provide the brain with foresight into the visual world.

Visual neurons respond selectively to features that become increasingly complex from the eyes to the cortex. Retinal neurons prefer flashing spots of light, primary visual cortical (V1) neurons prefer moving bars, and those in higher cortical areas favor complex features like moving textures. Previously, we showed that V1 simple cell tuning can be accounted for by a basic model implementing temporal prediction – representing features that predict future sensory input from past input (Singer et al., 2018). Here, we show that hierarchical application of temporal prediction can capture how tuning properties change across at least two levels of the visual system. This suggests that the brain does not efficiently represent all incoming information; instead, it selectively represents sensory inputs that help in predicting the future. When applied hierarchically, temporal prediction extracts time-varying features that depend on increasingly high-level statistics of the sensory input.

The cochlea decomposes incoming sound waveforms into different frequency components along the length of its basilar membrane. The receptor hair cells at the apical end of this resonant membrane are tuned to the lowest sound frequencies, with the preferred sound frequency of hair cell tuning increasing near-exponentially along the length of the membrane towards its basal end. This frequency composition of the sound is then transmitted to the brain by the auditory nerve fibers that innervate the inner hair cells. Hair cells respond to a sound impulse with a temporally asymmetric envelope and the sharpness of their tuning changes as the frequency to which they are most sensitive varies with their position along the basilar membrane. We ask if there is a normative explanation for why the cochlea decomposes sounds in this manner. Inspired by findings in the retina, we propose that cochlear tuning properties may be optimized for temporal prediction. This principle states that the sensory features represented by neurons are optimized to predict immediate future input from recent past input. We show that an artificial neural network optimized for temporal prediction of the immediate future of raw waveforms of natural sounds from their recent past produces tuning properties that resemble those observed in the auditory nerve. Specifically, the model captures the temporally asymmetric impulse responses, the tonotopic distribution and variation in tuning sharpness along the cochlea, and the frequency glide polarity of the impulse responses. These characteristics are not captured by a similar model optimized for compression of the sound waveform, rather than prediction. Given its success in accounting for the tuning properties at various processing levels in the auditory and visual systems, this finding for the cochlea provides further evidence that temporal prediction may be a general principle of sensory processing.