Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit
Preprint
Abstract
In deep latent Gaussian models, the latent variable is generated by a time-inhomogeneous Markov chain, where at each time step we pass the current state through a parametric nonlinear map, such as a feedforward neural net, and add a small independent Gaussian perturbation. This work considers the diffusion limit of such models, where the number of layers tends to infinity, while the step size and the noise variance tend to zero. The limiting latent object is an It\^o diffusion process that solves a stochastic differential equation (SDE) whose drift and diffusion coefficient are implemented by neural nets. We develop a variational inference framework for these \textit{neural SDEs} via stochastic automatic differentiation in Wiener space, where the variational approximations to the posterior are obtained by Girsanov (mean-shift) transformation of the standard Wiener process and the computation of gradients is based on the theory of stochastic flows. This permits the use of black-box SDE solvers and automatic differentiation for end-to-end inference. Experimental results with synthetic data are provided.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Convergence of the deep BSDE method for coupled FBSDEsProbability, Uncertainty and Quantitative Risk, 2020
- A probabilistic model for the numerical solution of initial value problemsStatistics and Computing, 2018
- Stable architectures for deep neural networksInverse Problems, 2017
- Variational Inference: A Review for StatisticiansJournal of the American Statistical Association, 2017
- Statistical analysis of differential equations: introducing probability measures on numerical solutionsStatistics and Computing, 2016
- Adaptive time-stepping for the strong numerical solution of stochastic differential equationsNumerical Algorithms, 2014
- Numerical Algorithms for Forward-Backward Stochastic Differential EquationsSIAM Journal on Scientific Computing, 2006
- Sensitivity Analysis Using Itô--Malliavin Calculus and Martingales, and Application to Stochastic Optimal ControlSIAM Journal on Control and Optimization, 2005
- A Monte Carlo EM Approach for Partially Observable Diffusion Processes: Theory and Applications to Neural NetworksNeural Computation, 2002
- Direct and adjoint sensitivity equations for parameter optimizationIEEE Transactions on Automatic Control, 1967