JZJacob A. Zavatone-Veth

Jacob A. Zavatone-Veth

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Jacob A. Zavatone-Veth has a total of 47 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Risk and cross validation in ridge regression with correlated samples *

    by Alexander Atanasov, Jacob A Zavatone-Veth, Cengiz Pehlevan

    Recent years have seen substantial advances in our understanding of high-dimensional ridge regression, but existing theories assume that training examples are independent. By leveraging techniques from random matrix theory and free probability, we provide sharp asymptotics for the in- and out-of-sample risks of ridge regression when the data points have arbitrary correlations. We demonstrate that in this setting, the generalized cross validation estimator (GCV) fails to correctly predict the out-of-sample risk. However, in the case where the noise residuals have the same correlations as the data points, one can modify the GCV to yield an efficiently-computable unbiased estimator that concentrates in the high-dimensional limit, which we dub CorrGCV. We further extend our asymptotic analysis to the case where the test point has nontrivial correlations with the training set, a setting often encountered in time series forecasting. Assuming knowledge of the correlation structure of the time series, this again yields an extension of the GCV estimator, and sharply characterizes the degree to which such test points yield an overly optimistic prediction of long-time risk. We validate the predictions of our theory across a variety of high-dimensional data.


  • Scaling and renormalization in high-dimensional regression

    by Alexander Atanasov, Jacob A Zavatone-Veth, Cengiz Pehlevan

    From benign overfitting in overparameterized models to rich power-law scalings in performance, simple ridge regression displays surprising behaviors sometimes thought to be limited to deep neural networks. This balance of phenomenological richness with analytical tractability makes ridge regression the model system of choice in high-dimensional machine learning. In this paper, we present a unifying perspective on recent results on ridge regression using the basic tools of random matrix theory and free probability, aimed at readers with backgrounds in physics and deep learning. We highlight the fact that statistical fluctuations in empirical covariance matrices can be absorbed into a renormalization of the ridge parameter. This “deterministic equivalence” allows us to obtain analytic formulas for the training and generalization errors in a few lines of algebra by leveraging the properties of the S -transform of free probability. From these precise asymptotics, we can easily identify sources of power-law scaling in model performance. In all models, the S -transform corresponds to the train-test generalization gap, and yields an analog of the generalized-cross-validation estimator. Using these techniques, we derive fine-grained bias-variance decompositions for a very general class of random feature models with structured covariates. This allows us to discover a scaling regime for random feature models where the variance due to the features limits performance in the overparameterized setting. We also demonstrate how anisotropic weight structure in random feature models can limit performance and lead to nontrivial exponents for finite-width corrections in the overparameterized setting. Our results extend and provide a unifying perspective on earlier models of neural scaling laws.


  • Open Access

    Convergent motifs of early olfactory processing are recapitulated by layer-wise efficient coding

    by Juan Carlos Fernández del Castillo, Farhad Pashakhanloo, Venkatesh N. Murthy, Jacob A. Zavatone-Veth

    The architecture of early olfactory processing is a striking example of convergent evolution. Typically, a panel of broadly tuned receptors is selectively expressed in sensory neurons (each neuron expressing only one receptor), and each glomerulus receives projections from just one neuron type. Taken together, these three motifs—broad receptors, selective expression, and glomerular convergence—constitute “canonical olfaction,” since a number of model organisms including mice and flies exhibit these features. The emergence of this distinctive architecture across evolutionary lineages suggests that it may be optimized for information processing, an idea known as efficient coding. In this work, we show that by maximizing mutual information one layer at a time, efficient coding recovers several features of canonical olfactory processing under realistic biophysical assumptions. We also explore the settings in which noncanonical olfaction may be advantageous. Along the way, we make several predictions relating olfactory circuits to features of receptor families and the olfactory environment.


  • Open Access

    Structure, disorder, and dynamics in task-trained recurrent neural circuits

    by David G. Clark, Blake Bordelon, Jacob A. Zavatone-Veth, Cengiz Pehlevan

    Across many brain areas, neurons produce heterogeneous, seemingly disordered responses. Yet such circuits cannot be purely random, since they must possess some structure to generate the representations and computations underlying behavior. How much structure is present in recurrent connectivity relative to disorder, and how the interaction between the two shapes population dynamics and single-neuron responses, remain incompletely understood. Recurrent neural networks trained to perform tasks have become a leading model of such circuits, but conventional training yields a single point in a vast space of task-compatible solutions, with no systematic way to explore this space and no theory of how internal representations vary within it. Without such a theory, the questions above cannot be addressed, and comparisons between trained networks and neural data are difficult to interpret. Here, we introduce a control parameter that governs the degree to which learning reshapes recurrent connectivity, interpolating between a reservoir regime and one in which recurrent weights are restructured by learning to produce task-relevant internal representations. Varying this parameter generates a family of task-compatible solutions whose internal dynamics differ in a controlled and interpretable way. We derive a dynamical mean-field theory showing that, while population-level dynamics converge to a deterministic limit, individual neurons are driven by independent samples from a single-neuron input-current distribution. When connectivity is random, this distribution is Gaussian. Recurrent restructuring drives it toward task-dependent, non-Gaussian forms. In linear networks, restructuring amplifies task-relevant frequencies. In nonlinear networks, it drives a phase transition from chaotic, high-dimensional activity to ordered, low-dimensional dynamics that generalize temporally beyond the training period. We apply the theory to a reaching task in which a recurrent network must reproduce macaque muscle activity, and find that optimally matching simultaneous motor-cortex recordings requires only a small degree of restructuring, with learned structure coexisting with random heterogeneity. These results suggest a broader picture in which large recurrent circuits are largely random but contain, to varying degrees, structured recurrent connectivity sufficient for generalizable, task-relevant representations. 1


  • Open Access

    A Random Matrix Theory Perspective on the Consistency of Diffusion Models

    by Binxu Wang, Jacob Zavatone-Veth, Cengiz Pehlevan

    Diffusion models trained on different, non-overlapping subsets of a dataset often produce strikingly similar outputs when given the same noise seed. We trace this consistency to a simple linear effect: the shared Gaussian statistics across splits already predict much of the generated images. To formalize this, we develop a random matrix theory (RMT) framework that quantifies how finite datasets shape the expectation and variance of the learned denoiser and sampling map in the linear setting. For expectations, sampling variability acts as a renormalization of the noise level through a self-consistent relation $σ^2 \mapsto κ(σ^2)$, explaining why limited data overshrink low-variance directions and pull samples toward the dataset mean. For fluctuations, our variance formulas reveal three key factors behind cross-split disagreement: \textit{anisotropy} across eigenmodes, \textit{inhomogeneity} across inputs, and overall scaling with dataset size. Extending deterministic-equivalence tools to fractional matrix powers further allows us to analyze entire sampling trajectories. The theory sharply predicts the behavior of linear diffusion models, and we validate its predictions on UNet and DiT architectures in their non-memorization regime, identifying where and how samples deviates across training data split. This provides a principled baseline for reproducibility in diffusion training, linking spectral properties of data to the stability of generative outputs.

  • Open Access

    A note on the dynamics of extended-context disordered kinetic spin models

    by Jacob Zavatone-Veth, Cengiz Pehlevan

    Inspired by striking advances in language modeling, there has recently been much interest in developing autogressive sequence models that are amenable to analytical study. In this short note, we consider extensions of simple disordered kinetic glass models from statistical physics. These models have tunable correlations, are easy to sample, and can be solved exactly when the state space dimension is large. In particular, we give an expository derivation of the dynamical mean field theories that describe their asymptotic statistics. We therefore propose that they constitute an interesting set of toy models for autoregressive sequence generation, in which one might study learning dynamics.

  • Open Access

    Simultaneous detection and estimation in olfactory sensing

    by Chen Jiang, Matthew Y. He, Venkatesh N. Murthy, Cengiz Pehlevan, Jacob A. Zavatone-Veth, Paul Masset

    The mammalian olfactory system shows an exceptional ability for rapid and accurate decoding of both the identity and concentration of odorants. Previous works have used the theory of compressed sensing to elucidate the algorithmic basis for this capability: decoding odor information from the responses of a restricted repertoire of receptors is possible because only a few relevant odorants are present in any given sensory scene. However, existing circuit models for olfactory decoding still cannot contend with the complexity of naturalistic olfactory scenes; they are limited to detection of a handful of odorants. Here, we propose a model for olfactory compressed sensing inspired by simultaneous localization and mapping algorithms in navigation, in which the set of present odors and their concentrations are inferred separately. We implement this split inference in a biologically-plausible recurrent circuit by introducing separate dynamics matched to the distinct nature of presence and concentration, and drawing on the framework of Mirrored Langevin Dynamics. This model can accurately infer presence and concentration at scale. Moreover, its circuit structure can be mapped onto the primary cell types of the olfactory bulb, giving a possible normative account for functional differences between mitral and tufted cells. Our approach offers a general path towards circuit algorithms for probabilistic inference—in olfactory sensing and beyond—that both perform well in naturalistic environments and make experimentally-testable predictions for neural response dynamics.

  • Open Access

    Pretrain-Test Task Alignment Governs Generalization in In-Context Learning

    by Mary I. Letey, Jacob A. Zavatone-Veth, Yue M. Lu, Cengiz Pehlevan

    In-context learning (ICL) is a central capability of Transformer models, but the structures in data that enable its emergence and govern its robustness remain poorly understood. In this work, we study how the structure of pretraining tasks governs generalization in ICL. Using a solvable model for ICL of linear regression by linear attention, we derive an exact expression for ICL generalization error in high dimensions under arbitrary pretraining-testing task covariance mismatch. This leads to a new alignment measure that quantifies how much information about the pretraining task distribution is useful for inference at test time. We show that this measure directly predicts ICL performance not only in the solvable model but also in nonlinear Transformers. Our analysis further reveals a tradeoff between specialization and generalization in ICL: depending on task distribution alignment, increasing pretraining task diversity can either improve or harm test performance. Together, these results identify train-test task alignment as a key determinant of generalization in ICL.

  • Open Access

    Convergent motifs of early olfactory processing are recapitulated by layer-wise efficient coding

    by Juan Carlos Fernández del Castillo, Farhad Pashakhanloo, Venkatesh N. Murthy, Jacob A. Zavatone-Veth

    The architecture of early olfactory processing is a striking example of convergent evolution. Typically, a panel of broadly tuned receptors is selectively expressed in sensory neurons (each neuron expressing only one receptor), and each glomerulus receives projections from just one neuron type. Taken together, these three motifs—broad receptors, selective expression, and glomerular convergence—constitute “canonical olfaction,” since a number of model organisms including mice and flies exhibit these features. The emergence of this distinctive architecture across evolutionary lineages suggests that it may be optimized for information processing, an idea known as efficient coding. In this work, we show that by maximizing mutual information one layer at a time, efficient coding recovers several features of canonical olfactory processing under realistic biophysical assumptions. We also explore the settings in which noncanonical olfaction may be advantageous. Along the way, we make several predictions relating olfactory circuits to features of receptor families and the olfactory environment.


  • Open Access

    Summary statistics of learning link changing neural representations to behavior

    by Jacob A. Zavatone-Veth, Blake Bordelon, Cengiz Pehlevan

    How can we make sense of large-scale recordings of neural activity across learning? Theories of neural network learning with their origins in statistical physics offer a potential answer: for a given task, there are often a small set of summary statistics that are sufficient to predict performance as the network learns. Here, we review recent advances in how summary statistics can be used to build theoretical understanding of neural network learning. We then argue for how this perspective can inform the analysis of neural data, enabling better understanding of learning in biological and artificial neural networks.