SGSurya Ganguli

Surya Ganguli

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Surya Ganguli has a total of 144 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Large-scale cellular-resolution read/write of activity enables discovery of cell types defined by complex circuit properties

    by Antonia Drinnenberg, Alexander Attinger, Allan Raventos, Charu Ramakrishnan, Barsin Eshaghi Gharagoz, Arjay Cordero, YoungJu Jo, La’akea Siverts, Tanya L. Daigle, Bosiljka Tasic, et al.

    SUMMARY: The complexity of the mammalian brain’s vast population of interconnected neurons poses a formidable challenge to elucidate its underlying mechanisms of coordination and computation. A key step forward will be technologies that can perform large-scale, cellular-resolution monitoring and interrogation of distributed brain circuit activity in behaving animals. Here, we present an all-optical strategy for precise optogenetic activity control of ∼10 3 neurons and simultaneous activity monitoring of ∼10 4 neurons within and across areas of mouse cortex—an order-of-magnitude leap beyond previous capabilities. Tracking population responses following delivery of precisely-defined widely-distributed activity patterns to the visual cortex of awake mice, we were surprised to identify neurons robustly responsive to stimulation of diverse ensembles, defying conventional like-to-like wiring rules. These cells were primarily deep L2/3 somatostatin-positive (SST) interneurons with functional properties distinct from other SST neurons, and appeared to play a role in brain dynamics that could only have been identified through broad cellular-resolution circuit interrogation. Our work reveals the value of measuring large-scale circuit-dynamical properties of functionally-resolved single cells, beyond genetic and anatomical classification, to define and explore the roles of cell types in brain function.


  • Open Access

    The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion

    by Daniel Kunin, Javier Sagastuy-Brena, Lauren Gillespie, Eshed Margalit, Hidenori Tanaka, Surya Ganguli, Daniel L. K. Yamins

    In this work we explore the limiting dynamics of deep neural networks trained with stochastic gradient descent (SGD). As observed previously, long after performance has converged, networks continue to move through parameter space by a process of anomalous diffusion in which distance travelled grows as a power law in the number of gradient updates with a nontrivial exponent. We reveal an intricate interaction between the hyperparameters of optimization, the structure in the gradient noise, and the Hessian matrix at the end of training that explains this anomalous diffusion. To build this understanding, we first derive a continuous-time model for SGD with finite learning rates and batch sizes as an underdamped Langevin equation. We study this equation in the setting of linear regression, where we can derive exact, analytic expressions for the phase space dynamics of the parameters and their instantaneous velocities from initialization to stationarity. Using the Fokker-Planck equation, we show that the key ingredient driving these dynamics is not the original training loss, but rather the combination of a modified loss, which implicitly regularizes the velocity, and probability currents, which cause oscillations in phase space. We identify qualitative and quantitative predictions of this theory in the dynamics of a ResNet-18 model trained on ImageNet. Through the lens of statistical physics, we uncover a mechanistic origin for the anomalous limiting dynamics of deep neural networks trained with SGD.

  • The Limiting Dynamics of SGD: Modified Loss, Phase-Space Oscillations, and Anomalous Diffusion

    by Daniel Kunin, Javier Sagastuy-Brena, Lauren Gillespie, Eshed Margalit, Hidenori Tanaka, Surya Ganguli, Daniel L. K. Yamins

    In this work, we explore the limiting dynamics of deep neural networks trained with stochastic gradient descent (SGD). As observed previously, long after performance has converged, networks continue to move through parameter space by a process of anomalous diffusion in which distance traveled grows as a power law in the number of gradient updates with a nontrivial exponent. We reveal an intricate interaction among the hyperparameters of optimization, the structure in the gradient noise, and the Hessian matrix at the end of training that explains this anomalous diffusion. To build this understanding, we first derive a continuous-time model for SGD with finite learning rates and batch sizes as an underdamped Langevin equation. We study this equation in the setting of linear regression, where we can derive exact, analytic expressions for the phase-space dynamics of the parameters and their instantaneous velocities from initialization to stationarity. Using the Fokker-Planck equation, we show that the key ingredient driving these dynamics is not the original training loss but rather the combination of a modified loss, which implicitly regularizes the velocity, and probability currents that cause oscillations in phase space. We identify qualitative and quantitative predictions of this theory in the dynamics of a ResNet-18 model trained on ImageNet. Through the lens of statistical physics, we uncover a mechanistic origin for the anomalous limiting dynamics of deep neural networks trained with SGD. Understanding the limiting dynamics of SGD, and its dependence on various important hyperparameters like batch size, learning rate, and momentum, can serve as a basis for future work that can turn these insights into algorithmic gains.


  • Open Access

    Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression

    by Allan Raventós, Mansheej Paul, Feng Chen, Surya Ganguli

    Pretrained transformers exhibit the remarkable ability of in-context learning (ICL): they can learn tasks from just a few examples provided in the prompt without updating any weights. This raises a foundational question: can ICL solve fundamentally $\textit{new}$ tasks that are very different from those seen during pretraining? To probe this question, we examine ICL's performance on linear regression while varying the diversity of tasks in the pretraining dataset. We empirically demonstrate a $\textit{task diversity threshold}$ for the emergence of ICL. Below this threshold, the pretrained transformer cannot solve unseen regression tasks, instead behaving like a Bayesian estimator with the $\textit{non-diverse pretraining task distribution}$ as the prior. Beyond this threshold, the transformer significantly outperforms this estimator; its behavior aligns with that of ridge regression, corresponding to a Gaussian prior over $\textit{all tasks}$, including those not seen during pretraining. Thus, when pretrained on data with task diversity greater than the threshold, transformers $\textit{can}$ optimally solve fundamentally new tasks in-context. Importantly, this capability hinges on it deviating from the Bayes optimal estimator with the pretraining distribution as the prior. This study also explores the effect of regularization, model capacity and task structure and underscores, in a concrete example, the critical role of task diversity, alongside data and model scale, in the emergence of ICL. Code is available at https://github.com/mansheej/icl-task-diversity.

  • Open Access

    One-shot entorhinal maps enable flexible navigation in novel environments

    by John H. Wen, Ben Sorscher, Surya Ganguli, Lisa M Giocomo

    Animals face the substantial challenge of navigating novel environments to find food, shelter, or mates. In mammals, hexagonal grid cells in the medial entorhinal cortex create a map-like population representation of the external environment 1–7 . However, it remains unclear how the grid map can adapt to novel environmental features on a rapid, behaviorally relevant time scale. By recording over fifteen thousand grid cells in mice navigating virtual environments, we found grid cell activity was weakly anchored to landmark inputs through a fixed circuit relationship. A computational model based on this fixed circuit assumption accurately predicted grid spatial patterns in environments with novel landmark rearrangements. Finally, a medial entorhinal cortex-dependent task revealed that while grid cell firing patterns remain anchored to landmarks, behavior can adapt to changes in landmark location via a downstream region implementing behavioral time scale synaptic plasticity 8 . This fixed but weak anchoring of grid cells to landmarks endows the grid map with powerful computational properties. The fixed nature allows the generation of rapid stable maps for novel environments after a single exposure. The weak nature allows these rapidly formed maps to incur only small distortions between distances traveled in real versus neural space. Overall, such rapid low distortion mapping can then mediate accurate navigational behavior in rapidly changing environments through downstream plasticity.


  • Open Access

    Information Geometry of the Retinal Representation Manifold

    by Xuehao Ding, Dongsoo Lee, Joshua B. Melander, George Sivulka, Surya Ganguli, Stephen A. Baccus

    The ability for the brain to discriminate among visual stimuli is constrained by their retinal representations. Previous studies of visual discriminability have been limited to either low-dimensional artificial stimuli or pure theoretical considerations without a realistic encoding model. Here we propose a novel framework for understanding stimulus discriminability achieved by retinal representations of naturalistic stimuli with the method of information geometry. To model the joint probability distribution of neural responses conditioned on the stimulus, we created a stochastic encoding model of a population of salamander retinal ganglion cells based on a three-layer convolutional neural network model. This model not only accurately captured the mean response to natural scenes but also a variety of second-order statistics. With the model and the proposed theory, we computed the Fisher information metric over stimuli to study the most discriminable stimulus directions. We found that the most discriminable stimulus varied substantially across stimuli, allowing an examination of the relationship between the most discriminable stimulus and the current stimulus. By examining responses generated by the most discriminable stimuli we further found that the most discriminative response mode is often aligned with the most stochastic mode. This finding carries the important implication that under natural scenes, retinal noise correlations are information-limiting rather than increasing information transmission as has been previously speculated. We additionally observed that sensitivity saturates less in the population than for single cells and that as a function of firing rate, Fisher information varies less than sensitivity. We conclude that under natural scenes, population coding benefits from complementary coding and helps to equalize the information carried by different firing rates, which may facilitate decoding of the stimulus under principles of information maximization.


  • Open Access

    Deep Learning on a Data Diet: Finding Important Examples Early in Training

    by Mansheej Paul, Surya Ganguli, Gintare Karolina Dziugaite

    Recent success in deep learning has partially been driven by training increasingly overparametrized networks on ever larger datasets. It is therefore natural to ask: how much of the data is superfluous, which examples are important for generalization, and how do we find them? In this work, we make the striking observation that, in standard vision datasets, simple scores averaged over several weight initializations can be used to identify important examples very early in training. We propose two such scores -- the Gradient Normed (GraNd) and the Error L2-Norm (EL2N) scores -- and demonstrate their efficacy on a range of architectures and datasets by pruning significant fractions of training data without sacrificing test accuracy. In fact, using EL2N scores calculated a few epochs into training, we can prune half of the CIFAR10 training set while slightly improving test accuracy. Furthermore, for a given dataset, EL2N scores from one architecture or hyperparameter configuration generalize to other configurations. Compared to recent work that prunes data by discarding examples that are rarely forgotten over the course of training, our scores use only local information early in training. We also use our scores to detect noisy examples and study training dynamics through the lens of important examples -- we investigate how the data distribution shapes the loss surface and identify subspaces of the model's data representation that are relatively stable over training.


  • Open Access

    The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks

    by Daniel Kunin, Atsushi Yamamura, Chao Ma, Surya Ganguli

    In this work, we explore the maximum-margin bias of quasi-homogeneous neural networks trained with gradient flow on an exponential loss and past a point of separability. We introduce the class of quasi-homogeneous models, which is expressive enough to describe nearly all neural networks with homogeneous activations, even those with biases, residual connections, and normalization layers, while structured enough to enable geometric analysis of its gradient dynamics. Using this analysis, we generalize the existing results of maximum-margin bias for homogeneous networks to this richer class of models. We find that gradient flow implicitly favors a subset of the parameters, unlike in the case of a homogeneous model where all parameters are treated equally. We demonstrate through simple examples how this strong favoritism toward minimizing an asymmetric norm can degrade the robustness of quasi-homogeneous models. On the other hand, we conjecture that this norm-minimization discards, when possible, unnecessary higher-order parameters, reducing the model to a sparser parameterization. Lastly, by applying our theorem to sufficiently expressive neural networks with normalization layers, we reveal a universal mechanism behind the empirical phenomenon of Neural Collapse.

  • Open Access

    When and why grid cells appear or not in trained path integrators

    by Ben Sorscher, Gabriel C. Mel, Aran Nayebi, Lisa Giocomo, Daniel Yamins, Surya Ganguli

    Recent work has claimed that the emergence of grid cells from trained path-integrator circuits is a more fragile phenomenon than previously reported. In this note we critically assess the main analysis and simulation results underlying this claim, within the proper context of previously published theoretical work. Our assessment reveals that the emergence of grid cells is entirely consistent with this prior theory: hexagonal grid cells robustly emerge precisely when prior theory predicts they should, and don’t when prior theory predicts they should not.