Daniel A. Roberts
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Daniel A. Roberts has a total of 16 co-authors. Here are the top co-authors by number of shared publications:
- Douglas Stanford
School of Natural Sciences Institute for Advanced Study Princeton NJ USA
4 shared publications - Yonatan Kahn
Department of Physics University of Toronto M5S 1A7 Toronto ON Canada
3 shared publications - Beni Yoshida
Perimeter Institute for Theoretical Physics
2 shared publications - Brian Swingle
Stanford Institute for Theoretical Physics Department of Physics Stanford University 94305 Stanford CA USA
2 shared publications - Jesse Thaler
Massachusetts Institute of Technology
2 shared publications - Joshua Batson
Chan Zuckerberg Biohub
2 shared publications
Latest Publications
- Open Access
- 7 August 2025
- Published by IOP Publishing in Machine Learning: Science and Technology
- Vol. 6 (3) , 035027
Feature learning and generalization in deep networks with orthogonal weights
Fully-connected deep neural networks with weights initialized from independent Gaussian distributions can be tuned to criticality, which prevents the exponential growth or decay of signals propagating through the network. However, such networks still exhibit fluctuations that grow linearly with the depth of the network, which may impair the training of networks with width comparable to depth. We show analytically that rectangular networks with tanh activations and weights initialized from the ensemble of orthogonal matrices have corresponding preactivation fluctuations which are independent of depth, to leading order in inverse width. Moreover, we demonstrate numerically that, at initialization, all correlators involving the neural tangent kernel (NTK) and its descendants at leading order in inverse width—which govern the evolution of observables during training—saturate at a depth of ∼ 20 , rather than growing without bound as in the case of Gaussian initializations. We speculate that this structure preserves finite-width feature learning while reducing overall noise, thus improving both generalization and training speed in deep networks with depth comparable to width. We provide some experimental justification by relating empirical measurements of the NTK to the superior performance of deep non-linear orthogonal networks trained under full-batch gradient descent on the MNIST and CIFAR-10 classification tasks.
- 7 August 2025
- Published by IOP Publishing in Machine Learning: Science and Technology
- Vol. 6 (3) , 035027
Open AccessFeature learning and generalization in deep networks with orthogonal weights
Fully-connected deep neural networks with weights initialized from independent Gaussian distributions can be tuned to criticality, which prevents the exponential growth or decay of signals propagating through the network. However, such networks still exhibit fluctuations that grow linearly with the depth of the network, which may impair the training of networks with width comparable to depth. We show analytically that rectangular networks with tanh activations and weights initialized from the ensemble of orthogonal matrices have corresponding preactivation fluctuations which are independent of depth, to leading order in inverse width. Moreover, we demonstrate numerically that, at initialization, all correlators involving the neural tangent kernel (NTK) and its descendants at leading order in inverse width—which govern the evolution of observables during training—saturate at a depth of ∼ 20 , rather than growing without bound as in the case of Gaussian initializations. We speculate that this structure preserves finite-width feature learning while reducing overall noise, thus improving both generalization and training speed in deep networks with depth comparable to width. We provide some experimental justification by relating empirical measurements of the NTK to the superior performance of deep non-linear orthogonal networks trained under full-batch gradient descent on the MNIST and CIFAR-10 classification tasks.
- Open Access
The Principles of Deep Learning Theory
This book develops an effective theory approach to understanding deep neural networks of practical relevance. Beginning from a first-principles component-level picture of networks, we explain how to determine an accurate description of the output of trained networks by solving layer-to-layer iteration equations and nonlinear learning dynamics. A main result is that the predictions of networks are described by nearly-Gaussian distributions, with the depth-to-width aspect ratio of the network controlling the deviations from the infinite-width Gaussian description. We explain how these effectively-deep networks learn nontrivial representations from training and more broadly analyze the mechanism of representation learning for nonlinear models. From a nearly-kernel-methods perspective, we find that the dependence of such models' predictions on the underlying learning algorithm can be expressed in a simple and universal way. To obtain these results, we develop the notion of representation group flow (RG flow) to characterize the propagation of signals through the network. By tuning networks to criticality, we give a practical solution to the exploding and vanishing gradient problem. We further explain how RG flow leads to near-universal behavior and lets us categorize networks built from different activation functions into universality classes. Altogether, we show that the depth-to-width ratio governs the effective model complexity of the ensemble of trained networks. By using information-theoretic techniques, we estimate the optimal aspect ratio at which we expect the network to be practically most useful and show how residual connections can be used to push this scale to arbitrary depths. With these tools, we can learn in detail about the inductive bias of architectures, hyperparameters, and optimizers.
Open AccessThe Principles of Deep Learning Theory
This book develops an effective theory approach to understanding deep neural networks of practical relevance. Beginning from a first-principles component-level picture of networks, we explain how to determine an accurate description of the output of trained networks by solving layer-to-layer iteration equations and nonlinear learning dynamics. A main result is that the predictions of networks are described by nearly-Gaussian distributions, with the depth-to-width aspect ratio of the network controlling the deviations from the infinite-width Gaussian description. We explain how these effectively-deep networks learn nontrivial representations from training and more broadly analyze the mechanism of representation learning for nonlinear models. From a nearly-kernel-methods perspective, we find that the dependence of such models' predictions on the underlying learning algorithm can be expressed in a simple and universal way. To obtain these results, we develop the notion of representation group flow (RG flow) to characterize the propagation of signals through the network. By tuning networks to criticality, we give a practical solution to the exploding and vanishing gradient problem. We further explain how RG flow leads to near-universal behavior and lets us categorize networks built from different activation functions into universality classes. Altogether, we show that the depth-to-width ratio governs the effective model complexity of the ensemble of trained networks. By using information-theoretic techniques, we estimate the optimal aspect ratio at which we expect the network to be practically most useful and show how residual connections can be used to push this scale to arbitrary depths. With these tools, we can learn in detail about the inductive bias of architectures, hyperparameters, and optimizers.
- Open Access
Topological Obstructions to Autoencoding
Autoencoders have been proposed as a powerful tool for model-independent anomaly detection in high-energy physics. The operating principle is that events which do not belong to the space of training data will be reconstructed poorly, thus flagging them as anomalies. We point out that in a variety of examples of interest, the connection between large reconstruction error and anomalies is not so clear. In particular, for data sets with nontrivial topology, there will always be points that erroneously seem anomalous due to global issues. Conversely, neural networks typically have an inductive bias or prior to locally interpolate such that undersampled or rare events may be reconstructed with small error, despite actually being the desired anomalies. Taken together, these facts are in tension with the simple picture of the autoencoder as an anomaly detector. Using a series of illustrative low-dimensional examples, we show explicitly how the intrinsic and extrinsic topology of the dataset affects the behavior of an autoencoder and how this topology is manifested in the latent space representation during training. We ground this analysis in the discussion of a mock "bump hunt" in which the autoencoder fails to identify an anomalous "signal" for reasons tied to the intrinsic topology of $n$-particle phase space.
Open AccessTopological Obstructions to Autoencoding
Autoencoders have been proposed as a powerful tool for model-independent anomaly detection in high-energy physics. The operating principle is that events which do not belong to the space of training data will be reconstructed poorly, thus flagging them as anomalies. We point out that in a variety of examples of interest, the connection between large reconstruction error and anomalies is not so clear. In particular, for data sets with nontrivial topology, there will always be points that erroneously seem anomalous due to global issues. Conversely, neural networks typically have an inductive bias or prior to locally interpolate such that undersampled or rare events may be reconstructed with small error, despite actually being the desired anomalies. Taken together, these facts are in tension with the simple picture of the autoencoder as an anomaly detector. Using a series of illustrative low-dimensional examples, we show explicitly how the intrinsic and extrinsic topology of the dataset affects the behavior of an autoencoder and how this topology is manifested in the latent space representation during training. We ground this analysis in the discussion of a mock "bump hunt" in which the autoencoder fails to identify an anomalous "signal" for reasons tied to the intrinsic topology of $n$-particle phase space.
- Open Access
- 29 April 2021
- Published by Springer Nature in Journal of High Energy Physics
- Vol. 2021 (4) , 1-43
Topological obstructions to autoencoding
Autoencoders have been proposed as a powerful tool for model-independent anomaly detection in high-energy physics. The operating principle is that events which do not belong to the space of training data will be reconstructed poorly, thus flagging them as anomalies. We point out that in a variety of examples of interest, the connection between large reconstruction error and anomalies is not so clear. In particular, for data sets with nontrivial topology, there will always be points that erroneously seem anomalous due to global issues. Conversely, neural networks typically have an inductive bias or prior to locally interpolate such that undersampled or rare events may be reconstructed with small error, despite actually being the desired anomalies. Taken together, these facts are in tension with the simple picture of the autoencoder as an anomaly detector. Using a series of illustrative low-dimensional examples, we show explicitly how the intrinsic and extrinsic topology of the dataset affects the behavior of an autoencoder and how this topology is manifested in the latent space representation during training. We ground this analysis in the discussion of a mock “bump hunt” in which the autoencoder fails to identify an anomalous “signal” for reasons tied to the intrinsic topology of n-particle phase space.
- 29 April 2021
- Published by Springer Nature in Journal of High Energy Physics
- Vol. 2021 (4) , 1-43
Open AccessTopological obstructions to autoencoding
Autoencoders have been proposed as a powerful tool for model-independent anomaly detection in high-energy physics. The operating principle is that events which do not belong to the space of training data will be reconstructed poorly, thus flagging them as anomalies. We point out that in a variety of examples of interest, the connection between large reconstruction error and anomalies is not so clear. In particular, for data sets with nontrivial topology, there will always be points that erroneously seem anomalous due to global issues. Conversely, neural networks typically have an inductive bias or prior to locally interpolate such that undersampled or rare events may be reconstructed with small error, despite actually being the desired anomalies. Taken together, these facts are in tension with the simple picture of the autoencoder as an anomaly detector. Using a series of illustrative low-dimensional examples, we show explicitly how the intrinsic and extrinsic topology of the dataset affects the behavior of an autoencoder and how this topology is manifested in the latent space representation during training. We ground this analysis in the discussion of a mock “bump hunt” in which the autoencoder fails to identify an anomalous “signal” for reasons tied to the intrinsic topology of n-particle phase space.
- Top Cited PapersOpen Access
- 23 August 2016
- Published by American Physical Society (APS) in Physical Review Letters
- Vol. 117 (9) , 091602
Lieb-Robinson Bound and the Butterfly Effect in Quantum Field Theories
As experiments are increasingly able to probe the quantum dynamics of systems with many degrees of freedom, it is interesting to probe fundamental bounds on the dynamics of quantum information. We elaborate on the relationship between one such bound—the Lieb-Robinson bound—and the butterfly effect in strongly coupled quantum systems. The butterfly effect implies the ballistic growth of local operators in time, which can be quantified with the “butterfly” velocity . Similarly, the Lieb-Robinson velocity places a state-independent ballistic upper bound on the size of time evolved operators in nonrelativistic lattice models. Here, we argue that is a state-dependent effective Lieb-Robinson velocity. We study the butterfly velocity in a wide variety of quantum field theories using holography and compare with free-particle computations to understand the role of strong coupling. We find that remains constant or decreases with decreasing temperature. We also comment on experimental prospects and on the relationship between the butterfly velocity and signaling.- 23 August 2016
- Published by American Physical Society (APS) in Physical Review Letters
- Vol. 117 (9) , 091602
Top Cited PapersOpen AccessLieb-Robinson Bound and the Butterfly Effect in Quantum Field Theories
As experiments are increasingly able to probe the quantum dynamics of systems with many degrees of freedom, it is interesting to probe fundamental bounds on the dynamics of quantum information. We elaborate on the relationship between one such bound—the Lieb-Robinson bound—and the butterfly effect in strongly coupled quantum systems. The butterfly effect implies the ballistic growth of local operators in time, which can be quantified with the “butterfly” velocity . Similarly, the Lieb-Robinson velocity places a state-independent ballistic upper bound on the size of time evolved operators in nonrelativistic lattice models. Here, we argue that is a state-dependent effective Lieb-Robinson velocity. We study the butterfly velocity in a wide variety of quantum field theories using holography and compare with free-particle computations to understand the role of strong coupling. We find that remains constant or decreases with decreasing temperature. We also comment on experimental prospects and on the relationship between the butterfly velocity and signaling. - Open Access
Two-dimensional conformal field theory and the butterfly effect
We study chaotic dynamics in two-dimensional conformal field theory through out-of-time order thermal correlators of the form $\langle W(t)VW(t)V\rangle$. We reproduce bulk calculations similar to those of [1], by studying the large $c$ Virasoro identity block. The contribution of this block to the above correlation function begins to decrease exponentially after a delay of $\sim t_* - \frac{\beta}{2\pi}\log \beta^2E_w E_v$, where $t_*$ is the scrambling time $\frac{\beta}{2\pi}\log c$, and $E_w,E_v$ are the energy scales of the $W,V$ operators.Open AccessTwo-dimensional conformal field theory and the butterfly effect
We study chaotic dynamics in two-dimensional conformal field theory through out-of-time order thermal correlators of the form $\langle W(t)VW(t)V\rangle$. We reproduce bulk calculations similar to those of [1], by studying the large $c$ Virasoro identity block. The contribution of this block to the above correlation function begins to decrease exponentially after a delay of $\sim t_* - \frac{\beta}{2\pi}\log \beta^2E_w E_v$, where $t_*$ is the scrambling time $\frac{\beta}{2\pi}\log c$, and $E_w,E_v$ are the energy scales of the $W,V$ operators. - Open Access
Lieb-Robinson and the butterfly effect
As experiments are increasingly able to probe the quantum dynamics of systems with many degrees of freedom, it is interesting to probe fundamental bounds on the dynamics of quantum information. We elaborate on the relationship between one such bound---the Lieb-Robinson bound---and the butterfly effect in strongly-coupled quantum systems. The butterfly effect implies the ballistic growth of local operators in time, which can be quantified with the "butterfly" velocity $v_B$. Similarly, the Lieb-Robinson velocity places a state independent ballistic upper bound on the size of time evolved operators in non-relativistic lattice models. Here, we argue that $v_B$ is a state-dependent effective Lieb-Robinson velocity. We study the butterfly velocity in a wide variety of quantum field theories using holography and compare with free particle computations to understand the role of strong coupling. We find that, depending on the way length and time scale, $v_B$ acquires a temperature dependence and decreases towards the IR. We also comment on experimental prospects and on the relationship between the butterfly velocity and signaling.Open AccessLieb-Robinson and the butterfly effect
As experiments are increasingly able to probe the quantum dynamics of systems with many degrees of freedom, it is interesting to probe fundamental bounds on the dynamics of quantum information. We elaborate on the relationship between one such bound---the Lieb-Robinson bound---and the butterfly effect in strongly-coupled quantum systems. The butterfly effect implies the ballistic growth of local operators in time, which can be quantified with the "butterfly" velocity $v_B$. Similarly, the Lieb-Robinson velocity places a state independent ballistic upper bound on the size of time evolved operators in non-relativistic lattice models. Here, we argue that $v_B$ is a state-dependent effective Lieb-Robinson velocity. We study the butterfly velocity in a wide variety of quantum field theories using holography and compare with free particle computations to understand the role of strong coupling. We find that, depending on the way length and time scale, $v_B$ acquires a temperature dependence and decreases towards the IR. We also comment on experimental prospects and on the relationship between the butterfly velocity and signaling. - Open Access
Localized shocks
We study products of precursors of spatially local operators, $W_{x_{n}}(t_{n}) ... W_{x_1}(t_1)$, where $W_x(t) = e^{-iHt} W_x e^{iHt}$. Using chaotic spin-chain numerics and gauge/gravity duality, we show that a single precursor fills a spatial region that grows linearly in $t$. In a lattice system, products of such operators can be represented using tensor networks. In gauge/gravity duality, they are related to Einstein-Rosen bridges supported by localized shock waves. We find a geometrical correspondence between these two descriptions, generalizing earlier work in the spatially homogeneous case.Open AccessLocalized shocks
We study products of precursors of spatially local operators, $W_{x_{n}}(t_{n}) ... W_{x_1}(t_1)$, where $W_x(t) = e^{-iHt} W_x e^{iHt}$. Using chaotic spin-chain numerics and gauge/gravity duality, we show that a single precursor fills a spatial region that grows linearly in $t$. In a lattice system, products of such operators can be represented using tensor networks. In gauge/gravity duality, they are related to Einstein-Rosen bridges supported by localized shock waves. We find a geometrical correspondence between these two descriptions, generalizing earlier work in the spatially homogeneous case. - Open Access
Chaos in quantum channels
We study chaos and scrambling in unitary channels by considering their entanglement properties as states. Using out-of-time-order correlation functions to diagnose chaos, we characterize the ability of a channel to process quantum information. We show that the generic decay of such correlators implies that any input subsystem must have near vanishing mutual information with almost all partitions of the output. Additionally, we propose the negativity of the tripartite information of the channel as a general diagnostic of scrambling. This measures the delocalization of information and is closely related to the decay of out-of-time-order correlators. We back up our results with numerics in two non-integrable models and analytic results in a perfect tensor network model of chaotic time evolution. These results show that the butterfly effect in quantum systems implies the information-theoretic definition of scrambling.
Open AccessChaos in quantum channels
We study chaos and scrambling in unitary channels by considering their entanglement properties as states. Using out-of-time-order correlation functions to diagnose chaos, we characterize the ability of a channel to process quantum information. We show that the generic decay of such correlators implies that any input subsystem must have near vanishing mutual information with almost all partitions of the output. Additionally, we propose the negativity of the tripartite information of the channel as a general diagnostic of scrambling. This measures the delocalization of information and is closely related to the decay of out-of-time-order correlators. We back up our results with numerics in two non-integrable models and analytic results in a perfect tensor network model of chaotic time evolution. These results show that the butterfly effect in quantum systems implies the information-theoretic definition of scrambling.
- Top Cited PapersOpen Access
- 1 February 2016
- Published by Springer Nature in Journal of High Energy Physics
- Vol. 2016 (2) , 1-49
Chaos in quantum channels
- 1 February 2016
- Published by Springer Nature in Journal of High Energy Physics
- Vol. 2016 (2) , 1-49
Top Cited PapersOpen AccessChaos in quantum channels