Friedemann Zenke

The retina’s role in visual processing has been viewed as two extremes: an efficient compressor of incoming visual stimuli, akin to a camera, or a predictor of future stimuli. Addressing this dichotomy, we developed a spiking neural network model of the retina trained on natural movies under metabolic-like constraints to either encode the present or to predict future scenes. When optimized for efficient temporal prediction ~100 ms into the future, the model not only captures retina-like receptive fields and their mosaic-like organizations, but also exhibits complex retinal processes such as latency coding, motion anticipation, differential motion tuning, and stimulus-omission responses. Notably, the temporal prediction model also accurately predicts the way retinal ganglion cells respond across different animal species to natural images and movies. Our findings suggest that the retina is not merely a compressor of visual input, but rather is fundamentally organized to provide the brain with foresight into the visual world.

Tracking prey or recognizing a lurking predator is as crucial for survival as anticipating their actions. To guide behavior, the brain must extract information about object identities and their dynamics from entangled sensory inputs. How it accomplishes this feat remains an open question. Classical predictive coding theories propose that this ability arises by comparing predicted sensory signals with actual inputs and reducing the associated prediction errors. While such models capture important aspects of cortical computation, they typically focus on faithfully predicting sensory input and do not explicitly address how abstract, untangled representations of objects and their dynamics emerge solely through experience. Here, we develop a theory of representation learning in neural circuits that shifts the focus from prediction in the input space to prediction in representation space, without relying on external supervision or labeled data. Specifically, we introduce recurrent predictive learning (RPL), a recurrent joint-embedding predictive architecture inspired by self-supervised machine learning, that learns abstract representations of object identity and their dynamics and predicts future object motion from continuous sensory streams. Crucially, the model learns sequence representations that resemble successor-like representations observed in the primary visual cortex of humans. The model also develops abstract sequence representations comparable to those reported in the macaque prefrontal cortex. Finally, we outline how RPL’s modular feedforward-recurrent organization could map onto cortical microcircuits. Our work establishes a circuit-centric theory framework that provides new perspectives on how the brain may acquire an internal model of the world through experience.

Tracking prey or recognizing a lurking predator is as crucial for survival as anticipating their actions. To guide behavior, the brain must extract information about object identities and their dynamics from entangled sensory inputs. How it accomplishes this feat remains an open question. Classical predictive coding theories propose that this ability arises by comparing predicted sensory signals with actual inputs and reducing the associated prediction errors. While such models capture important aspects of cortical computation, they typically focus on faithfully predicting sensory input and do not explicitly address how abstract, untangled representations of objects and their dynamics emerge solely through experience. Here, we develop a theory of representation learning in neural circuits that shifts the focus from prediction in the input space to prediction in representation space, without relying on external supervision or labeled data. Specifically, we introduce recurrent predictive learning (RPL), a recurrent joint-embedding predictive architecture inspired by self-supervised machine learning, that learns abstract representations of object identity and their dynamics and predicts future object motion from continuous sensory streams. Crucially, the model learns sequence representations that resemble successor-like representations observed in the primary visual cortex of humans. The model also develops abstract sequence representations comparable to those reported in the macaque prefrontal cortex. Finally, we outline how RPL’s modular feedforward-recurrent organization could map onto cortical microcircuits. Our work establishes a circuit-centric theory framework that provides new perspectives on how the brain may acquire an internal model of the world through experience.

Generative biology holds the promise to transform our ability to design and understand living systems by creating novel proteins, pathways, and organisms with tailored functions that address challenges in medicine, sustainability, and technology. However, training generative models requires large quantities of data that captures the genetic architecture of protein function in all its complexity, but these are currently scarce. Here, we systematically mutagenized all 54 human basic-leucine zipper (bZIP) domains and quantified their interactions with each other using bindingPCA, a quantitative deep mutational scanning assay. This resulted in ∼2 million interaction measurements, capturing the effect of all single amino acid substitutions at each of the 35 interfacial positions. We found that mutation effects are largely additive in the vicinity of each wild-type bZIPs, but diverge across the family, indicating strong context dependency. A global additive thermodynamic model provided moderate prediction of mutation effects, while individual models per bZIP achieved higher performance, supporting a model of local simplicity and global complexity. Our results therefore suggest that the genetic architecture of protein function is more complex than previously anticipated, which could hinder predictability. However, a convolutional neural network trained on this dataset could accurately predict binding scores from sequence alone. Furthermore, the model enabled the design of synthetic bZIPs with high target specificity, demonstrating practical applicability for bioengineering purposes. Our study shows that capturing family-wide diversity is essential to reveal context dependencies and train accurate quantitative models of protein-protein interactions.

Deploying energy-efficient deep neural networks on energy-constrained edge devices is an important research topic in both machine learning and circuit design communities. Both ANNs and SNNs have been proposed as candidates for these tasks and in particular SNNs have been proposed as energy-efficient because they leverage temporal sparsity in the outputs. However, existing computational frameworks fail to accurately estimate the cost of running sparse networks on modern time-stepped hardware, which exploits sparsity by skipping zero-valued operations. Meanwhile, weight sparsity-aware training remains underexplored for SNN and lacks systematic benchmarking against optimized ANN, making fair comparisons between the two paradigms difficult. To bridge this gap, we introduce EFLOP, a metric that accounts for the sparse operations during pre-activation updates of both ANN and SNN. Applying weight sparsity-aware training to both SNN and ANN, we achieve up to 8.9x reduction in EFLOP for GRU models and 3.6x for LIF models by sparsifying weights by 80%, without sacrificing accuracy on the SHD and SSC datasets. Our findings highlight the critical role of network sparsity in designing energy-efficient neural networks and establish EFLOP as a robust framework for cross-paradigm comparisons.

Efficient learning algorithms such as backpropagation, predictive coding, and self-supervised learning require instructive learning signals to influence synaptic plasticity locally at specific neurons. This requirement is in-consistent with classic Hebbian theories and neuromodulation involving a non-specific third factor, thus raising the question of how neurobiology can achieve efficient learning. Here, we propose a simple and biologically plausible solution: local deviations from precise excitation/inhibition (E/I) balance encode error signals that instruct synaptic plasticity. Using a computational model derived from an adaptive control theory framework, we demonstrate that breaking E/I balance through targeted feedback to inhibitory interneurons, can produce neuron- or assembly-specific error signals that enable learning in multi-layer networks. Simulations reveal that such a balance-controlled plasticity mechanism is consistent with phenomenological local plasticity models while enabling online learning in hierarchically organized networks. Furthermore, we demonstrate that this framework is consistent with key features of disinhibitory microcircuit dynamics during in-vivo learning experiments. These results suggest that the brain may exploit E/I balance not only for stability, but also as a substrate for error-driven learning.

Synaptic plasticity is a key player in the brain’s life-long learning abilities. However, due to experimental limitations, the mechanistic link between synaptic plasticity rules and the network-level computations they enable remain opaque. Here we use evolutionary strategies (ES) to meta learn local co-active plasticity rules in large recurrent spiking networks with excitatory (E) and inhibitory (I) neurons, using parameterizations of increasing complexity. We discover rules that robustly stabilize network dynamics for all four synapse types acting in isolation (E-to-E, E-to-I, I-to-E and I-to-I). More complex functions such as familiarity detection can also be included in the search constraints. However, our meta learning strategy begins to fail for co-active rules of increasing complexity, as it is challenging to devise loss functions that effectively constrain network dynamics to plausible solutions a priori. Moreover, in line with previous work, we can find multiple degenerate solutions with identical network behaviour. As a local optimization strategy, ES provides one solution at a time and makes exploration of this degeneracy cumbersome. Regardless, we can glean the interdependecies of various plasticity parameters by considering the covariance matrix learned alongside the optimal rule with ES. Our work provides a proof of principle for the success of machine-learning-guided discovery of plasticity rules in large spiking networks, and points at the necessity of more elaborate search strategies going forward.

Training spiking neural networks to approximate universal functions is essential for studying information processing in the brain and for neuromorphic computing. Yet the binary nature of spikes poses a challenge for direct gradient-based training. Surrogate gradients have been empirically successful in circumventing this problem, but their theoretical foundation remains elusive. Here, we investigate the relation of surrogate gradients to two theoretically well-founded approaches. On the one hand, we consider smoothed probabilistic models, which, due to the lack of support for automatic differentiation, are impractical for training multilayer spiking neural networks but provide derivatives equivalent to surrogate gradients for single neurons. On the other hand, we investigate stochastic automatic differentiation, which is compatible with discrete randomness but has not yet been used to train spiking neural networks. We find that the latter gives surrogate gradients a theoretical basis in stochastic spiking neural networks, where the surrogate derivative matches the derivative of the neuronal escape noise function. This finding supports the effectiveness of surrogate gradients in practice and suggests their suitability for stochastic spiking neural networks. However, surrogate gradients are generally not gradients of a surrogate loss despite their relation to stochastic automatic differentiation. Nevertheless, we empirically confirm the effectiveness of surrogate gradients in stochastic multilayer spiking neural networks and discuss their relation to deterministic networks as a special case. Our work gives theoretical support to surrogate gradients and the choice of a suitable surrogate derivative in stochastic spiking neural networks.

Biological memory networks are thought to store information by experience-dependent changes in the synaptic connectivity between assemblies of neurons. Recent models suggest that these assemblies contain both excitatory and inhibitory neurons (E/I assemblies), resulting in co-tuning and precise balance of excitation and inhibition. To understand computational consequences of E/I assemblies under biologically realistic constraints we built a spiking network model based on experimental data from telencephalic area Dp of adult zebrafish, a precisely balanced recurrent network homologous to piriform cortex. We found that E/I assemblies stabilized firing rate distributions compared to networks with excitatory assemblies and global inhibition. Unlike classical memory models, networks with E/I assemblies did not show discrete attractor dynamics. Rather, responses to learned inputs were locally constrained onto manifolds that ‘focused’ activity into neuronal subspaces. The covariance structure of these manifolds supported pattern classification when information was retrieved from selected neuronal subsets. Networks with E/I assemblies therefore transformed the geometry of neuronal coding space, resulting in continuous representations that reflected both relatedness of inputs and an individual’s experience. Such continuous representations enable fast pattern classification, can support continual learning, and may provide a basis for higher-order learning and cognitive computations.