DRDaniel A. Roberts

Daniel A. Roberts

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Daniel A. Roberts has a total of 16 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Feature learning and generalization in deep networks with orthogonal weights

    by Hannah Day, Yonatan Kahn, Daniel A Roberts

    Fully-connected deep neural networks with weights initialized from independent Gaussian distributions can be tuned to criticality, which prevents the exponential growth or decay of signals propagating through the network. However, such networks still exhibit fluctuations that grow linearly with the depth of the network, which may impair the training of networks with width comparable to depth. We show analytically that rectangular networks with tanh activations and weights initialized from the ensemble of orthogonal matrices have corresponding preactivation fluctuations which are independent of depth, to leading order in inverse width. Moreover, we demonstrate numerically that, at initialization, all correlators involving the neural tangent kernel (NTK) and its descendants at leading order in inverse width—which govern the evolution of observables during training—saturate at a depth of ∼ 20 , rather than growing without bound as in the case of Gaussian initializations. We speculate that this structure preserves finite-width feature learning while reducing overall noise, thus improving both generalization and training speed in deep networks with depth comparable to width. We provide some experimental justification by relating empirical measurements of the NTK to the superior performance of deep non-linear orthogonal networks trained under full-batch gradient descent on the MNIST and CIFAR-10 classification tasks.


  • Open Access

    The Principles of Deep Learning Theory

    by Daniel A. Roberts, Sho Yaida, Boris Hanin

    This book develops an effective theory approach to understanding deep neural networks of practical relevance. Beginning from a first-principles component-level picture of networks, we explain how to determine an accurate description of the output of trained networks by solving layer-to-layer iteration equations and nonlinear learning dynamics. A main result is that the predictions of networks are described by nearly-Gaussian distributions, with the depth-to-width aspect ratio of the network controlling the deviations from the infinite-width Gaussian description. We explain how these effectively-deep networks learn nontrivial representations from training and more broadly analyze the mechanism of representation learning for nonlinear models. From a nearly-kernel-methods perspective, we find that the dependence of such models' predictions on the underlying learning algorithm can be expressed in a simple and universal way. To obtain these results, we develop the notion of representation group flow (RG flow) to characterize the propagation of signals through the network. By tuning networks to criticality, we give a practical solution to the exploding and vanishing gradient problem. We further explain how RG flow leads to near-universal behavior and lets us categorize networks built from different activation functions into universality classes. Altogether, we show that the depth-to-width ratio governs the effective model complexity of the ensemble of trained networks. By using information-theoretic techniques, we estimate the optimal aspect ratio at which we expect the network to be practically most useful and show how residual connections can be used to push this scale to arbitrary depths. With these tools, we can learn in detail about the inductive bias of architectures, hyperparameters, and optimizers.


  • Open Access

    Topological Obstructions to Autoencoding

    by Joshua Batson, C. Grace Haaf, Yonatan Kahn, Daniel A. Roberts

    Autoencoders have been proposed as a powerful tool for model-independent anomaly detection in high-energy physics. The operating principle is that events which do not belong to the space of training data will be reconstructed poorly, thus flagging them as anomalies. We point out that in a variety of examples of interest, the connection between large reconstruction error and anomalies is not so clear. In particular, for data sets with nontrivial topology, there will always be points that erroneously seem anomalous due to global issues. Conversely, neural networks typically have an inductive bias or prior to locally interpolate such that undersampled or rare events may be reconstructed with small error, despite actually being the desired anomalies. Taken together, these facts are in tension with the simple picture of the autoencoder as an anomaly detector. Using a series of illustrative low-dimensional examples, we show explicitly how the intrinsic and extrinsic topology of the dataset affects the behavior of an autoencoder and how this topology is manifested in the latent space representation during training. We ground this analysis in the discussion of a mock "bump hunt" in which the autoencoder fails to identify an anomalous "signal" for reasons tied to the intrinsic topology of $n$-particle phase space.

  • Open Access

    Topological obstructions to autoencoding

    by Joshua Batson, C. Grace Haaf, Yonatan Kahn, Daniel A. Roberts

    Autoencoders have been proposed as a powerful tool for model-independent anomaly detection in high-energy physics. The operating principle is that events which do not belong to the space of training data will be reconstructed poorly, thus flagging them as anomalies. We point out that in a variety of examples of interest, the connection between large reconstruction error and anomalies is not so clear. In particular, for data sets with nontrivial topology, there will always be points that erroneously seem anomalous due to global issues. Conversely, neural networks typically have an inductive bias or prior to locally interpolate such that undersampled or rare events may be reconstructed with small error, despite actually being the desired anomalies. Taken together, these facts are in tension with the simple picture of the autoencoder as an anomaly detector. Using a series of illustrative low-dimensional examples, we show explicitly how the intrinsic and extrinsic topology of the dataset affects the behavior of an autoencoder and how this topology is manifested in the latent space representation during training. We ground this analysis in the discussion of a mock “bump hunt” in which the autoencoder fails to identify an anomalous “signal” for reasons tied to the intrinsic topology of n-particle phase space.


  • Top Cited Papers
    Open Access

    Lieb-Robinson Bound and the Butterfly Effect in Quantum Field Theories

    by Daniel A. Roberts, Brian Swingle

    As experiments are increasingly able to probe the quantum dynamics of systems with many degrees of freedom, it is interesting to probe fundamental bounds on the dynamics of quantum information. We elaborate on the relationship between one such bound—the Lieb-Robinson bound—and the butterfly effect in strongly coupled quantum systems. The butterfly effect implies the ballistic growth of local operators in time, which can be quantified with the “butterfly” velocity vB. Similarly, the Lieb-Robinson velocity places a state-independent ballistic upper bound on the size of time evolved operators in nonrelativistic lattice models. Here, we argue that vB is a state-dependent effective Lieb-Robinson velocity. We study the butterfly velocity in a wide variety of quantum field theories using holography and compare with free-particle computations to understand the role of strong coupling. We find that vB remains constant or decreases with decreasing temperature. We also comment on experimental prospects and on the relationship between the butterfly velocity and signaling.


  • Open Access

    Two-dimensional conformal field theory and the butterfly effect

    by Daniel A. Roberts, Douglas Stanford

    We study chaotic dynamics in two-dimensional conformal field theory through out-of-time order thermal correlators of the form $\langle W(t)VW(t)V\rangle$. We reproduce bulk calculations similar to those of [1], by studying the large $c$ Virasoro identity block. The contribution of this block to the above correlation function begins to decrease exponentially after a delay of $\sim t_* - \frac{\beta}{2\pi}\log \beta^2E_w E_v$, where $t_*$ is the scrambling time $\frac{\beta}{2\pi}\log c$, and $E_w,E_v$ are the energy scales of the $W,V$ operators.


  • Open Access

    Lieb-Robinson and the butterfly effect

    by Daniel A. Roberts, Brian Swingle

    As experiments are increasingly able to probe the quantum dynamics of systems with many degrees of freedom, it is interesting to probe fundamental bounds on the dynamics of quantum information. We elaborate on the relationship between one such bound---the Lieb-Robinson bound---and the butterfly effect in strongly-coupled quantum systems. The butterfly effect implies the ballistic growth of local operators in time, which can be quantified with the "butterfly" velocity $v_B$. Similarly, the Lieb-Robinson velocity places a state independent ballistic upper bound on the size of time evolved operators in non-relativistic lattice models. Here, we argue that $v_B$ is a state-dependent effective Lieb-Robinson velocity. We study the butterfly velocity in a wide variety of quantum field theories using holography and compare with free particle computations to understand the role of strong coupling. We find that, depending on the way length and time scale, $v_B$ acquires a temperature dependence and decreases towards the IR. We also comment on experimental prospects and on the relationship between the butterfly velocity and signaling.

  • Open Access

    Localized shocks

    by Daniel A. Roberts, Douglas Stanford, Leonard Susskind

    We study products of precursors of spatially local operators, $W_{x_{n}}(t_{n}) ... W_{x_1}(t_1)$, where $W_x(t) = e^{-iHt} W_x e^{iHt}$. Using chaotic spin-chain numerics and gauge/gravity duality, we show that a single precursor fills a spatial region that grows linearly in $t$. In a lattice system, products of such operators can be represented using tensor networks. In gauge/gravity duality, they are related to Einstein-Rosen bridges supported by localized shock waves. We find a geometrical correspondence between these two descriptions, generalizing earlier work in the spatially homogeneous case.


  • Open Access

    Chaos in quantum channels

    by Pavan Hosur, Xiao-Liang Qi, Daniel A. Roberts, Beni Yoshida

    We study chaos and scrambling in unitary channels by considering their entanglement properties as states. Using out-of-time-order correlation functions to diagnose chaos, we characterize the ability of a channel to process quantum information. We show that the generic decay of such correlators implies that any input subsystem must have near vanishing mutual information with almost all partitions of the output. Additionally, we propose the negativity of the tripartite information of the channel as a general diagnostic of scrambling. This measures the delocalization of information and is closely related to the decay of out-of-time-order correlators. We back up our results with numerics in two non-integrable models and analytic results in a perfect tensor network model of chaotic time evolution. These results show that the butterfly effect in quantum systems implies the information-theoretic definition of scrambling.

  • Top Cited Papers
    Open Access

    Chaos in quantum channels

    by Pavan Hosur, Xiao-Liang Qi, Daniel A. Roberts, Beni Yoshida