Soon Hoe Lim
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Soon Hoe Lim has a total of 22 co-authors. Here are the top co-authors by number of shared publications:
- Jan Wehr
Department of Mathematics and Program in Applied Mathematics, University of Arizona, Tucson, USA
4 shared publications Maciej Lewenstein
The Barcelona Institute of Science and Technology
4 shared publicationsJan Wehr
University of Arizona
3 shared publicationsAniello Lampo
The Barcelona Institute of Science and Technology
2 shared publications- Maciej Lewenstein
ICFO
2 shared publications - Michael W. Mahoney
Department of Statistics ICSI LBNL University of California at Berkeley
2 shared publications
Latest Publications
- Open Access
Gated Recurrent Neural Networks with Weighted Time-Delay Feedback
We introduce a novel gated recurrent unit (GRU) with a weighted time-delay feedback mechanism in order to improve the modeling of long-term dependencies in sequential data. This model is a discretized version of a continuous-time formulation of a recurrent unit, where the dynamics are governed by delay differential equations (DDEs). By considering a suitable time-discretization scheme, we propose $\tau$-GRU, a discrete-time gated recurrent unit with delay. We prove the existence and uniqueness of solutions for the continuous-time model, and we demonstrate that the proposed feedback mechanism can help improve the modeling of long-term dependencies. Our empirical results show that $\tau$-GRU can converge faster and generalize better than state-of-the-art recurrent units and gated recurrent architectures on a range of tasks, including time-series classification, human activity recognition, and speech recognition.
Open AccessGated Recurrent Neural Networks with Weighted Time-Delay Feedback
We introduce a novel gated recurrent unit (GRU) with a weighted time-delay feedback mechanism in order to improve the modeling of long-term dependencies in sequential data. This model is a discretized version of a continuous-time formulation of a recurrent unit, where the dynamics are governed by delay differential equations (DDEs). By considering a suitable time-discretization scheme, we propose $\tau$-GRU, a discrete-time gated recurrent unit with delay. We prove the existence and uniqueness of solutions for the continuous-time model, and we demonstrate that the proposed feedback mechanism can help improve the modeling of long-term dependencies. Our empirical results show that $\tau$-GRU can converge faster and generalize better than state-of-the-art recurrent units and gated recurrent architectures on a range of tasks, including time-series classification, human activity recognition, and speech recognition.
- Open Access
Chaotic Regularization and Heavy-Tailed Limits for Deterministic Gradient Descent
Recent studies have shown that gradient descent (GD) can achieve improved generalization when its dynamics exhibits a chaotic behavior. However, to obtain the desired effect, the step-size should be chosen sufficiently large, a task which is problem dependent and can be difficult in practice. In this study, we incorporate a chaotic component to GD in a controlled manner, and introduce multiscale perturbed GD (MPGD), a novel optimization framework where the GD recursion is augmented with chaotic perturbations that evolve via an independent dynamical system. We analyze MPGD from three different angles: (i) By building up on recent advances in rough paths theory, we show that, under appropriate assumptions, as the step-size decreases, the MPGD recursion converges weakly to a stochastic differential equation (SDE) driven by a heavy-tailed L\'evy-stable process. (ii) By making connections to recently developed generalization bounds for heavy-tailed processes, we derive a generalization bound for the limiting SDE and relate the worst-case generalization error over the trajectories of the process to the parameters of MPGD. (iii) We analyze the implicit regularization effect brought by the dynamical regularization and show that, in the weak perturbation regime, MPGD introduces terms that penalize the Hessian of the loss function. Empirical results are provided to demonstrate the advantages of MPGD.
Open AccessChaotic Regularization and Heavy-Tailed Limits for Deterministic Gradient Descent
Recent studies have shown that gradient descent (GD) can achieve improved generalization when its dynamics exhibits a chaotic behavior. However, to obtain the desired effect, the step-size should be chosen sufficiently large, a task which is problem dependent and can be difficult in practice. In this study, we incorporate a chaotic component to GD in a controlled manner, and introduce multiscale perturbed GD (MPGD), a novel optimization framework where the GD recursion is augmented with chaotic perturbations that evolve via an independent dynamical system. We analyze MPGD from three different angles: (i) By building up on recent advances in rough paths theory, we show that, under appropriate assumptions, as the step-size decreases, the MPGD recursion converges weakly to a stochastic differential equation (SDE) driven by a heavy-tailed L\'evy-stable process. (ii) By making connections to recently developed generalization bounds for heavy-tailed processes, we derive a generalization bound for the limiting SDE and relate the worst-case generalization error over the trajectories of the process to the parameters of MPGD. (iii) We analyze the implicit regularization effect brought by the dynamical regularization and show that, in the weak perturbation regime, MPGD introduces terms that penalize the Hessian of the loss function. Empirical results are provided to demonstrate the advantages of MPGD.
- Open Access
Noisy Feature Mixup
We introduce Noisy Feature Mixup (NFM), an inexpensive yet effective method for data augmentation that combines the best of interpolation based training and noise injection schemes. Rather than training with convex combinations of pairs of examples and their labels, we use noise-perturbed convex combinations of pairs of data points in both input and feature space. This method includes mixup and manifold mixup as special cases, but it has additional advantages, including better smoothing of decision boundaries and enabling improved model robustness. We provide theory to understand this as well as the implicit regularization effects of NFM. Our theory is supported by empirical results, demonstrating the advantage of NFM, as compared to mixup and manifold mixup. We show that residual networks and vision transformers trained with NFM have favorable trade-offs between predictive accuracy on clean data and robustness with respect to various types of data perturbation across a range of computer vision benchmark datasets.
Open AccessNoisy Feature Mixup
We introduce Noisy Feature Mixup (NFM), an inexpensive yet effective method for data augmentation that combines the best of interpolation based training and noise injection schemes. Rather than training with convex combinations of pairs of examples and their labels, we use noise-perturbed convex combinations of pairs of data points in both input and feature space. This method includes mixup and manifold mixup as special cases, but it has additional advantages, including better smoothing of decision boundaries and enabling improved model robustness. We provide theory to understand this as well as the implicit regularization effects of NFM. Our theory is supported by empirical results, demonstrating the advantage of NFM, as compared to mixup and manifold mixup. We show that residual networks and vision transformers trained with NFM have favorable trade-offs between predictive accuracy on clean data and robustness with respect to various types of data perturbation across a range of computer vision benchmark datasets.
- Open Access
- 22 March 2021
- Published by IOP Publishing in Journal of Physics A: Mathematical and Theoretical
- Vol. 54 (15) , 155001
Anomalous thermodynamics in homogenized generalized Langevin systems
We study functionals, such as heat and work, along trajectories of a class of multi-dimensional generalized Langevin systems in various limiting situations that correspond to different level of homogenization. These are the situations where one or more of the inertial time scale(s), the memory time scale(s) and the noise correlation time scale(s) of the systems are taken to zero. We find that, unless one restricts to special situations that do not break symmetry of the Onsager matrix associated with the fast dynamics, it is generally not possible to express the effective evolution of these functionals solely in terms of trajectory of the homogenized process describing the system dynamics via the widely adopted Stratonovich convention. In fact, an anomalous term is often needed for a complete description, implying that convergence of these functionals needs more information than simply the limit of the dynamical process. We trace the origin of such impossibility to area anomaly, thereby linking the symmetry breaking and area anomaly. This hold important consequences for many nonequilibrium systems that can be modeled by generalized Langevin equations. Our convergence results hold in a strong pathwise sense.
- 22 March 2021
- Published by IOP Publishing in Journal of Physics A: Mathematical and Theoretical
- Vol. 54 (15) , 155001
Open AccessAnomalous thermodynamics in homogenized generalized Langevin systems
We study functionals, such as heat and work, along trajectories of a class of multi-dimensional generalized Langevin systems in various limiting situations that correspond to different level of homogenization. These are the situations where one or more of the inertial time scale(s), the memory time scale(s) and the noise correlation time scale(s) of the systems are taken to zero. We find that, unless one restricts to special situations that do not break symmetry of the Onsager matrix associated with the fast dynamics, it is generally not possible to express the effective evolution of these functionals solely in terms of trajectory of the homogenized process describing the system dynamics via the widely adopted Stratonovich convention. In fact, an anomalous term is often needed for a complete description, implying that convergence of these functionals needs more information than simply the limit of the dynamical process. We trace the origin of such impossibility to area anomaly, thereby linking the symmetry breaking and area anomaly. This hold important consequences for many nonequilibrium systems that can be modeled by generalized Langevin equations. Our convergence results hold in a strong pathwise sense.
- Open Access
Anomalous Thermodynamics in Homogenized Generalized Langevin Systems
We study functionals, such as heat and work, along trajectories of a class of multi-dimensional generalized Langevin systems in various limiting situations that correspond to different level of homogenization. These are the situations where one or more of the inertial time scale(s), the memory time scale(s) and the noise correlation time scale(s) of the systems are taken to zero. We find that, unless one restricts to special situations that do not break symmetry of the Onsager matrix associated with the fast dynamics, it is generally not possible to express the effective evolution of these functionals solely in terms of trajectory of the homogenized process describing the system dynamics via the widely adopted Stratonovich convention. In fact, an anomalous term is often needed for a complete description, implying that convergence of these functionals needs more information than simply the limit of the dynamical process. We trace the origin of such impossibility to area anomaly, thereby linking the symmetry breaking and area anomaly. This hold important consequences for many nonequilibrium systems that can be modeled by generalized Langevin equations. Our convergence results hold in a strong pathwise sense.
Open AccessAnomalous Thermodynamics in Homogenized Generalized Langevin Systems
We study functionals, such as heat and work, along trajectories of a class of multi-dimensional generalized Langevin systems in various limiting situations that correspond to different level of homogenization. These are the situations where one or more of the inertial time scale(s), the memory time scale(s) and the noise correlation time scale(s) of the systems are taken to zero. We find that, unless one restricts to special situations that do not break symmetry of the Onsager matrix associated with the fast dynamics, it is generally not possible to express the effective evolution of these functionals solely in terms of trajectory of the homogenized process describing the system dynamics via the widely adopted Stratonovich convention. In fact, an anomalous term is often needed for a complete description, implying that convergence of these functionals needs more information than simply the limit of the dynamical process. We trace the origin of such impossibility to area anomaly, thereby linking the symmetry breaking and area anomaly. This hold important consequences for many nonequilibrium systems that can be modeled by generalized Langevin equations. Our convergence results hold in a strong pathwise sense.
- Open Access
Understanding Recurrent Neural Networks Using Nonequilibrium Response Theory
Recurrent neural networks (RNNs) are brain-inspired models widely used in machine learning for analyzing sequential data. The present work is a contribution towards a deeper understanding of how RNNs process input signals using the response theory from nonequilibrium statistical mechanics. For a class of continuous-time stochastic RNNs (SRNNs) driven by an input signal, we derive a Volterra type series representation for their output. This representation is interpretable and disentangles the input signal from the SRNN architecture. The kernels of the series are certain recursively defined correlation functions with respect to the unperturbed dynamics that completely determine the output. Exploiting connections of this representation and its implications to rough paths theory, we identify a universal feature -- the response feature, which turns out to be the signature of tensor product of the input signal and a natural support basis. In particular, we show that SRNNs, with only the weights in the readout layer optimized and the weights in the hidden layer kept fixed and not optimized, can be viewed as kernel machines operating on a reproducing kernel Hilbert space associated with the response feature.
Open AccessUnderstanding Recurrent Neural Networks Using Nonequilibrium Response Theory
Recurrent neural networks (RNNs) are brain-inspired models widely used in machine learning for analyzing sequential data. The present work is a contribution towards a deeper understanding of how RNNs process input signals using the response theory from nonequilibrium statistical mechanics. For a class of continuous-time stochastic RNNs (SRNNs) driven by an input signal, we derive a Volterra type series representation for their output. This representation is interpretable and disentangles the input signal from the SRNN architecture. The kernels of the series are certain recursively defined correlation functions with respect to the unperturbed dynamics that completely determine the output. Exploiting connections of this representation and its implications to rough paths theory, we identify a universal feature -- the response feature, which turns out to be the signature of tensor product of the input signal and a natural support basis. In particular, we show that SRNNs, with only the weights in the readout layer optimized and the weights in the hidden layer kept fixed and not optimized, can be viewed as kernel machines operating on a reproducing kernel Hilbert space associated with the response feature.
- Open Access
- 1 December 2020
- Published by AIP Publishing in Chaos: An Interdisciplinary Journal of Nonlinear Science
- Vol. 30 (12) , 123126
Predicting critical transitions in multiscale dynamical systems using reservoir computing
We study the problem of predicting rare critical transition events for a class of slow–fast nonlinear dynamical systems. The state of the system of interest is described by a slow process, whereas a faster process drives its evolution and induces critical transitions. By taking advantage of recent advances in reservoir computing, we present a data-driven method to predict the future evolution of the state. We show that our method is capable of predicting a critical transition event at least several numerical time steps in advance. We demonstrate the success as well as the limitations of our method using numerical experiments on three examples of systems, ranging from low dimensional to high dimensional. We discuss the mathematical and broader implications of our results.
- 1 December 2020
- Published by AIP Publishing in Chaos: An Interdisciplinary Journal of Nonlinear Science
- Vol. 30 (12) , 123126
Open AccessPredicting critical transitions in multiscale dynamical systems using reservoir computing
We study the problem of predicting rare critical transition events for a class of slow–fast nonlinear dynamical systems. The state of the system of interest is described by a slow process, whereas a faster process drives its evolution and induces critical transitions. By taking advantage of recent advances in reservoir computing, we present a data-driven method to predict the future evolution of the state. We show that our method is capable of predicting a critical transition event at least several numerical time steps in advance. We demonstrate the success as well as the limitations of our method using numerical experiments on three examples of systems, ranging from low dimensional to high dimensional. We discuss the mathematical and broader implications of our results.
- Open Access
- 8 February 2020
- Published by Springer Nature in Annales Henri Poincare
- Vol. 21 (6) , 1813-1871
Homogenization for Generalized Langevin Equations with Applications to Anomalous Diffusion
We study homogenization for a class of generalized Langevin equations (GLEs) with state-dependent coefficients and exhibiting multiple time scales. In addition to the small mass limit, we focus on homogenization limits, which involve taking to zero the inertial time scale and, possibly, some of the memory time scales and noise correlation time scales. The latter are meaningful limits for a class of GLEs modeling anomalous diffusion. We find that, in general, the limiting stochastic differential equations for the slow degrees of freedom contain non-trivial drift correction terms and are driven by non-Markov noise processes. These results follow from a general homogenization theorem stated and proven here. We illustrate them using stochastic models of particle diffusion.
- 8 February 2020
- Published by Springer Nature in Annales Henri Poincare
- Vol. 21 (6) , 1813-1871
Open AccessHomogenization for Generalized Langevin Equations with Applications to Anomalous Diffusion
We study homogenization for a class of generalized Langevin equations (GLEs) with state-dependent coefficients and exhibiting multiple time scales. In addition to the small mass limit, we focus on homogenization limits, which involve taking to zero the inertial time scale and, possibly, some of the memory time scales and noise correlation time scales. The latter are meaningful limits for a class of GLEs modeling anomalous diffusion. We find that, in general, the limiting stochastic differential equations for the slow degrees of freedom contain non-trivial drift correction terms and are driven by non-Markov noise processes. These results follow from a general homogenization theorem stated and proven here. We illustrate them using stochastic models of particle diffusion.
- Open Access
Homogenization for Generalized Langevin Equations with Applications to Anomalous Diffusion
We study homogenization for a class of generalized Langevin equations (GLEs) with state-dependent coefficients and exhibiting multiple time scales. In addition to the small mass limit, we focus on homogenization limits, which involve taking to zero the inertial time scale and, possibly, some of the memory time scales and noise correlation time scales. The latter are meaningful limits for a class of GLEs modeling anomalous diffusion. We find that, in general, the limiting stochastic differential equations (SDEs) for the slow degrees of freedom contain non-trivial drift correction terms and are driven by non-Markov noise processes. These results follow from a general homogenization theorem stated and proven here. We illustrate them using stochastic models of particle diffusion.
Open AccessHomogenization for Generalized Langevin Equations with Applications to Anomalous Diffusion
We study homogenization for a class of generalized Langevin equations (GLEs) with state-dependent coefficients and exhibiting multiple time scales. In addition to the small mass limit, we focus on homogenization limits, which involve taking to zero the inertial time scale and, possibly, some of the memory time scales and noise correlation time scales. The latter are meaningful limits for a class of GLEs modeling anomalous diffusion. We find that, in general, the limiting stochastic differential equations (SDEs) for the slow degrees of freedom contain non-trivial drift correction terms and are driven by non-Markov noise processes. These results follow from a general homogenization theorem stated and proven here. We illustrate them using stochastic models of particle diffusion.
- Open Access
- 1 August 2019
- Published by IOP Publishing in Journal of Statistical Mechanics: Theory and Experiment
- Vol. 2019 (8) , 084005
Functionals in stochastic thermodynamics: how to interpret stochastic integrals
In stochastic thermodynamics standard concepts from macroscopic thermodynamics, such as heat, work, and entropy production, are generalized to small fluctuating systems by defining them on a trajectory-wise level. In Langevin systems with continuous state-space such definitions involve stochastic integrals along system trajectories, whose specific values depend on the discretization rule used to evaluate them (i.e. the ‘interpretation’ of the noise terms in the integral). Via a systematic mathematical investigation of this apparent dilemma, we corroborate the widely used standard interpretation of heat- and work-like functionals as Stratonovich integrals. We furthermore recapitulate the anomalies that are known to occur for entropy production in the presence of temperature gradients.
- 1 August 2019
- Published by IOP Publishing in Journal of Statistical Mechanics: Theory and Experiment
- Vol. 2019 (8) , 084005
Open AccessFunctionals in stochastic thermodynamics: how to interpret stochastic integrals
In stochastic thermodynamics standard concepts from macroscopic thermodynamics, such as heat, work, and entropy production, are generalized to small fluctuating systems by defining them on a trajectory-wise level. In Langevin systems with continuous state-space such definitions involve stochastic integrals along system trajectories, whose specific values depend on the discretization rule used to evaluate them (i.e. the ‘interpretation’ of the noise terms in the integral). Via a systematic mathematical investigation of this apparent dilemma, we corroborate the widely used standard interpretation of heat- and work-like functionals as Stratonovich integrals. We furthermore recapitulate the anomalies that are known to occur for entropy production in the presence of temperature gradients.