Saumil S. Patel

Sensory systems support generalization by representing features that persist under input variation; however, identifying the neuronal basis of these invariances remains difficult due to high-dimensional and nonlinear neural computations. Here we leverage the inception loop paradigm, iterating between large-scale recordings, predictive models and in silico experiments with in vivo verification, to characterize neuronal invariances in mouse primary visual cortex (V1). We synthesize varied exciting inputs (VEIs), dissimilar images that drive target neurons. These VEIs revealed a new bipartite invariance: one subfield encodes a shift-tolerant high-frequency texture and the other encodes a fixed low-frequency pattern. This division aligns with object boundaries defined by spatial frequency differences in highly activating images, suggesting a contribution to segmentation. Analysis of the MICrONS dataset revealed a hierarchy of excitatory neurons in mouse V1 layers 2/3: postsynaptic neurons exhibited greater invariance than their presynaptic inputs, while neurons with lower invariance formed more connections. Together, these results provide insights and scalable methodology for mapping neuronal invariances.

Deciphering the brain’s structure-function relationship is key to understanding the neuronal mechanisms underlying perception and cognition. The cortical column, a vertical organization of neurons with similar functions, is a classic example of primate neocortex structure-function organization. While columns have been identified in primary sensory areas using parametric stimuli, their prevalence across higher-level cortex is debated, particularly regarding complex tuning in natural image space. However, a key hurdle in identifying columns is characterizing the complex, nonlinear tuning of neurons to high-dimensional sensory inputs. Building on prior findings of topological organization for features like color and orientation, we investigate functional clustering in macaque visual area V4 in non-parametric natural image space, using large-scale recordings and deep learning–based analysis. We combined linear probe recordings with deep learning methods to systematically characterize the tuning of >1,200 V4 neurons using in silico synthesis of most exciting images (MEIs), followed by in vivo verification. Single V4 neurons exhibited MEIs containing complex features, including textures and shapes, and even high-level attributes with eye-like appearance. Neurons recorded on the same silicon probe, inserted orthogonal to the cortical surface, often exhibited similarities in their spatial feature selectivity, suggesting a degree of functional organization along the cortical depth. We quantified MEI similarity using human psychophysics and distances in a contrastive learning-derived embedding space. Moreover, the selectivity of the V4 neuronal population showed evidence of clustering into functional groups of shared feature selectivity. These functional groups showed parallels with the feature maps of units in artificial vision systems, suggesting potential shared encoding strategies. These results demonstrate the feasibility and scalability of deep learning–based functional characterization of neuronal selectivity in naturalistic visual contexts, offering a framework for quantitatively mapping cortical organization across multiple levels of the visual hierarchy.

Electrocorticography uses non-penetrating electrodes embedded in flexible substrates to record electrical activity from the surface of the brain. To use the technology to develop minimally invasive, high-bandwidth brain–computer interfaces, it will be necessary to improve the number of recording channels and the scalability of devices, which could be achieved by merging electrodes and electronics onto a single substrate. Here we report a 50-μm-thick, mechanically flexible micro-electrocorticography brain–computer interface that integrates a 256 × 256 array of electrodes, signal processing, data telemetry and wireless powering on a single complementary metal–oxide–semiconductor substrate. The device contains 65,536 recording electrodes, from which we can simultaneously record a selectable subset of up to 1,024 channels at a given time. Our chip is wirelessly powered, and when implanted below the dura, it can communicate bidirectionally with an external relay station outside the body. We show that the device can provide chronic, reliable recordings for up to two weeks in pigs and up to two months in behaving non-human primates from the somatosensory, motor and visual cortices, decoding brain signals at high spatiotemporal resolution.

Cortical prosthetic systems offer a promising path to restoring vision in blindness by stimulating neurons in the visual cortex to evoke visual percepts. To effectively encode visual information, however, it is essential to target neurons based on their functional encoding properties. Current stimulation protocols focus on retinotopic information, ignoring other key encoding properties, and therefore fail to reproduce complex visual percepts. We demonstrate that incorporating orientation selectivity alongside retinotopy to guide stimulation dramatically improves fidelity of the evoked activity to the underlying neural code. We propose a Bottlenecked Rotation-Equivariant CNN (BRCNN) and demonstrate that neural responses can be predicted to a large degree using only retinotopy and orientation preference. Using this model, we design a retinotopy-and-orientation aware stimulation protocol and validate it in a state-of-the-art large-scale optogenetic stimulation simulation framework of primary visual cortex. Our protocol elicits neural activity patterns exhibiting substantially higher correlation with natural vision responses compared to retinotopic-only approaches.

Deciphering the brain’s structure-function relationship is key to understanding the neuronal mechanisms underlying perception and cognition. The cortical column, a vertical organization of neurons with similar functions, is a classic example of primate neocortex structure-function organization. While columns have been identified in primary sensory areas using parametric stimuli, their prevalence across higher-level cortex is debated, particularly regarding complex tuning in natural image space. However, a key hurdle in identifying columns is characterizing the complex, nonlinear tuning of neurons to high-dimensional sensory inputs. Building on prior findings of topological organization for features like color and orientation, we investigate functional clustering in macaque visual area V4 in non-parametric natural image space, using large-scale recordings and deep learning–based analysis. We combined linear probe recordings with deep learning methods to systematically characterize the tuning of >1,200 V4 neurons using in silico synthesis of most exciting images (MEIs), followed by in vivo verification. Single V4 neurons exhibited MEIs containing complex features, including textures and shapes, and even high-level attributes with eye-like appearance. Neurons recorded on the same silicon probe, inserted orthogonal to the cortical surface, often exhibited similarities in their spatial feature selectivity, suggesting a degree of functional organization along the cortical depth. We quantified MEI similarity using human psychophysics and distances in a contrastive learning-derived embedding space. Moreover, the selectivity of the V4 neuronal population showed evidence of clustering into functional groups of shared feature selectivity. These functional groups showed parallels with the feature maps of units in artificial vision systems, suggesting potential shared encoding strategies. These results demonstrate the feasibility and scalability of deep learning–based functional characterization of neuronal selectivity in naturalistic visual contexts, offering a framework for quantitatively mapping cortical organization across multiple levels of the visual hierarchy.

The complexity of neural circuits makes it challenging to decipher the brain’s algorithms of intelligence. Recent break-throughs in deep learning have produced models that accurately simulate brain activity, enhancing our understanding of the brain’s computational objectives and neural coding. However, these models struggle to generalize beyond their training distribution, limiting their utility. The emergence of foundation models, trained on vast datasets, has introduced a new AI paradigm with remarkable generalization capabilities. We collected large amounts of neural activity from visual cortices of multiple mice and trained a foundation model to accurately predict neuronal responses to arbitrary natural videos. This model generalized to new mice with minimal training and successfully predicted responses across various new stimulus domains, such as coherent motion and noise patterns. It could also be adapted to new tasks beyond neural prediction, accurately predicting anatomical cell types, dendritic features, and neuronal connectivity within the MICrONS functional connectomics dataset. Our work is a crucial step toward building foundation brain models. As neuroscience accumulates larger, multi-modal datasets, foundation models will uncover statistical regularities, enabling rapid adaptation to new tasks and accelerating research.

A defining characteristic of intelligent systems, whether natural or artificial, is the ability to generalize and infer behaviorally relevant latent causes from high-dimensional sensory input, despite significant variations in the environment. To understand how brains achieve generalization, it is crucial to identify the features to which neurons respond selectively and invariantly. However, the high-dimensional nature of visual inputs, the non-linearity of information processing in the brain, and limited experimental time make it challenging to systematically characterize neuronal tuning and invariances, especially for natural stimuli. Here, we extended “inception loops” — a paradigm that iterates between large-scale recordings, neural predictive models, and in silico experiments followed by in vivo verification — to systematically characterize single neuron invariances in the mouse primary visual cortex. Using the predictive model we synthesized Diverse Exciting Inputs (DEIs), a set of inputs that differ substantially from each other while each driving a target neuron strongly, and verified these DEIs’ efficacy in vivo . We discovered a novel bipartite invariance: one portion of the receptive field encoded phase-invariant texturelike patterns, while the other portion encoded a fixed spatial pattern. Our analysis revealed that the division between the fixed and invariant portions of the receptive fields aligns with object boundaries defined by spatial frequency differences present in highly activating natural images. These findings suggest that bipartite invariance might play a role in segmentation by detecting texture-defined object boundaries, independent of the phase of the texture. We also replicated these bipartite DEIs in the functional connectomics MICrONs data set, which opens the way towards a circuit-level mechanistic understanding of this novel type of invariance. Our study demonstrates the power of using a data-driven deep learning approach to systematically characterize neuronal invariances. By applying this method across the visual hierarchy, cell types, and sensory modalities, we can decipher how latent variables are robustly extracted from natural scenes, leading to a deeper understanding of generalization.

We are now in the era of millimeter-scale electron microscopy (EM) volumes collected at nanometer resolution (Shapson-Coe et al., 2021; Consortium et al., 2021). Dense reconstruction of cellular compartments in these EM volumes has been enabled by recent advances in Machine Learning (ML) (Lee et al., 2017; Wu et al., 2021; Lu et al., 2021; Macrina et al., 2021). Automated segmentation methods produce exceptionally accurate reconstructions of cells, but post-hoc proofreading is still required to generate large connectomes free of merge and split errors. The elaborate 3-D meshes of neurons in these volumes contain detailed morphological information at multiple scales, from the diameter, shape, and branching patterns of axons and dendrites, down to the fine-scale structure of dendritic spines. However, extracting these features can require substantial effort to piece together existing tools into custom workflows. Building on existing open-source software for mesh manipulation, here we present “NEURD”, a software package that decomposes meshed neurons into compact and extensively-annotated graph representations. With these feature-rich graphs, we automate a variety of tasks such as state of the art automated proofreading of merge errors, cell classification, spine detection, axon-dendritic proximities, and other annotations. These features enable many downstream analyses of neural morphology and connectivity, making these massive and complex datasets more accessible to neuroscience researchers focused on a variety of scientific questions.

Understanding the relationship between circuit connectivity and function is crucial for uncovering how the brain implements computation. In the mouse primary visual cortex (V1), excitatory neurons with similar response properties are more likely to be synaptically connected, but previous studies have been limited to within V1, leaving much unknown about broader connectivity rules. In this study, we leverage the millimeter-scale MICrONS dataset to analyze synaptic connectivity and functional properties of individual neurons across cortical layers and areas. Our results reveal that neurons with similar responses are preferentially connected both within and across layers and areas — including feedback connections — suggesting the universality of the ‘like-to-like’ connectivity across the visual hierarchy. Using a validated digital twin model, we separated neuronal tuning into feature (what neurons respond to) and spatial (receptive field location) components. We found that only the feature component predicts fine-scale synaptic connections, beyond what could be explained by the physical proximity of axons and dendrites. We also found a higher-order rule where postsynaptic neuron cohorts downstream of individual presynaptic cells show greater functional similarity than predicted by a pairwise like-to-like rule. Notably, recurrent neural networks (RNNs) trained on a simple classification task develop connectivity patterns mirroring both pairwise and higher-order rules, with magnitude similar to those in the MICrONS data. Lesion studies in these RNNs reveal that disrupting ‘like-to-like’ connections has a significantly greater impact on performance compared to lesions of random connections. These findings suggest that these connectivity principles may play a functional role in sensory processing and learning, highlighting shared principles between biological and artificial systems.

A key role of sensory processing is integrating information across space. Neuronal responses in the visual system are influenced by both local features in the receptive field center and contextual information from the surround. While center-surround interactions have been extensively studied using simple stimuli like gratings, investigating these interactions with more complex, ecologically-relevant stimuli is challenging due to the high dimensionality of the stimulus space. We used large-scale neuronal recordings in mouse primary visual cortex to train convolutional neural network (CNN) models that accurately predicted center-surround interactions for natural stimuli. These models enabled us to synthesize surround stimuli that strongly suppressed or enhanced neuronal responses to the optimal center stimulus, as confirmed by in vivo experiments. In contrast to the common notion that congruent center and surround stimuli are suppressive, we found that excitatory surrounds appeared to complete spatial patterns in the center, while inhibitory surrounds disrupted them. We quantified this effect by demonstrating that CNN-optimized excitatory surround images have strong similarity in neuronal response space with surround images generated by extrapolating the statistical properties of the center, and with patches of natural scenes, which are known to exhibit high spatial correlations. Our findings cannot be explained by theories like redundancy reduction or predictive coding previously linked to contextual modulation in visual cortex. Instead, we demonstrated that a hierarchical probabilistic model incorporating Bayesian inference, and modulating neuronal responses based on prior knowledge of natural scene statistics, can explain our empirical results. We replicated these center-surround effects in the multi-area functional connectomics MICrONS dataset using natural movies as visual stimuli, which opens the way towards understanding circuit level mechanism, such as the contributions of lateral and feedback recurrent connections. Our data-driven modeling approach provides a new understanding of the role of contextual interactions in sensory processing and can be adapted across brain areas, sensory modalities, and species.