Simon Tavener

Neural networks (NN) can be divided into two broad categories, recurrent and non-recurrent. Both types of neural networks are popular and extensively studied, but they are often treated as distinct families of machine learning algorithms. In this position paper, we argue that there is a closer relationship between these two types of neural networks than is normally appreciated. We show that many common neural network models, such as Recurrent Neural Networks (RNN), Multi-Layer Perceptrons (MLP), and even deep multi-layer transformers, can all be represented as iterative maps. The close relationship between RNNs and other types of NNs should not be surprising. In particular, RNNs are known to be Turing complete, and therefore capable of representing any computable function (such as any other types of NNs), but herein we argue that the relationship runs deeper and is more practical than this. For example, RNNs are often thought to be more difficult to train than other types of NNs, with RNNs being plagued by issues such as vanishing or exploding gradients. However, as we demonstrate in this paper, MLPs, RNNs, and many other NNs lie on a continuum, and this perspective leads to several insights that illuminate both theoretical and practical aspects of NNs.

We develop an \textit{a posteriori} error analysis for a numerical estimate of the time at which a functional of the solution to a partial differential equation (PDE) first achieves a threshold value on a given time interval. This quantity of interest (QoI) differs from classical QoIs which are modeled as bounded linear (or nonlinear) functionals {of the solution}. Taylor's theorem and an adjoint-based \textit{a posteriori} analysis is used to derive computable and accurate error estimates in the case of semi-linear parabolic and hyperbolic PDEs. The accuracy of the error estimates is demonstrated through numerical solutions of the one-dimensional heat equation and linearized shallow water equations (SWE), representing parabolic and hyperbolic cases, respectively.

We construct a space-time parallel method for solving parabolic partial differential equations by coupling the Parareal algorithm in time with overlapping domain decomposition in space. The goal is to obtain a discretization consisting of "local" problems that can be solved on parallel computers efficiently. However, this introduces significant sources of error that must be evaluated. Reformulating the original Parareal algorithm as a variational method and implementing a finite element discretization in space enables an adjoint-based a posteriori error analysis to be performed. Through an appropriate choice of adjoint problems and residuals the error analysis distinguishes between errors arising due to the temporal and spatial discretizations, as well as between the errors arising due to incomplete Parareal iterations and incomplete iterations of the domain decomposition solver. We first develop an error analysis for the Parareal method applied to parabolic partial differential equations, and then refine this analysis to the case where the associated spatial problems are solved using overlapping domain decomposition. These constitute our Time Parallel Algorithm (TPA) and Space-Time Parallel Algorithm (STPA) respectively. Numerical experiments demonstrate the accuracy of the estimator for both algorithms and the iterations between distinct components of the error.

Classical a posteriori error analysis for differential equations quantifies the error in a Quantity of Interest (QoI) which is represented as a bounded linear functional of the solution. In this work we consider a posteriori error estimates of a quantity of interest that cannot be represented in this fashion, namely the time at which a threshold is crossed for the first time. We derive two representations for such errors and use an adjoint-based a posteriori approach to estimate unknown terms that appear in our representation. The first representation is based on linearizations using Taylor's Theorem. The second representation is obtained by implementing standard root-finding techniques. We provide several examples which demonstrate the accuracy of the methods. We then embed these error estimates within a framework to provide error bounds on a cumulative distribution function when parameters of the differential equations are uncertain.

1 Biological systems have evolved a degree of robustness with respect to perturbations in their environment and this capability is essential for their survival. In applications ranging from therapeutics to conservation, it is important to understand not only the sensitivity of biological systems to changes in their environments, but which features of these systems are necessary to achieve a given outcome. Mathematical models are increasingly employed to understand these mechanisms. Sensitivity analyses of such mathematical models provide insight into the responsiveness of the system when experimental manipulation is difficult. One common approach is to seek the probability distribution of the outputs of the system corresponding to a known distribution of inputs. By contrast, inverse sensitivity analysis determines the probability distribution of model inputs which produces a known distribution of outputs. The computational complexity of the methods used to conduct inverse sensitivity analyses for deterministic systems has limited their application to models with relatively few parameters. Here we describe a novel Markov Chain Monte Carlo method we call “Contour Monte Carlo”, which can be used to invert systems with a large number of parameters. We demonstrate the utility of this method by inverting a range of frequently-used deterministic models of biological systems, including the logistic growth equation, the Michaelis-Menten equation, and an SIR model of disease transmission with nine input parameters. We argue that the simplicity of our approach means it is amenable to a large class of problems of practical significance and, more generally, provides a probabilistic framework for understanding the inversion of deterministic models. 2 Author summary: Mathematical models of complex systems are constructed to provide insight into their underlying functioning. Statistical inversion can probe the often unobserved processes underlying biological systems, by proceeding from a given distribution of a model’s outputs (the aggregate “effects”) to a distribution over input parameters (the constituent “causes”). The process of inversion is well-defined for systems involving randomness and can be described by Bayesian inference. The inversion of a deterministic system, however, cannot be performed by the standard Bayesian approach. We develop a conceptual framework that describes the inversion of deterministic systems with fewer outputs than input parameters. Like Bayesian inference, our approach uses probability distributions to describe the uncertainty over inputs and outputs, and requires a prior input distribution to ensure a unique “posterior” probability distribution over inputs. We describe a computational Monte Carlo method that allows efficient sampling from the posterior distribution even as the dimension of the input parameter space grows. This is a two-step process where we first estimate a “contour volume density” associated with each output value which is then used to define a sampling algorithm that yields the requisite input distribution asymptotically. Our approach is simple, broadly applicable and could be widely adopted.

We construct a stabilized finite-element method to compute flow and finite-strain deformations in an incompressible poroelastic medium. We employ a three-field mixed formulation to calculate displacement, fluid flux and pressure directly and introduce a Lagrange multiplier to enforce flux boundary conditions. We use a low order approximation, namely, continuous piecewise-linear approximation for the displacements and fluid flux, and piecewise-constant approximation for the pressure. This results in a simple matrix structure with low bandwidth. The method is stable in both the limiting cases of small and large permeability. Moreover, the discontinuous pressure space enables efficient approximation of steep gradients such as those occurring due to rapidly changing material coefficients or boundary conditions, both of which are commonly seen in physical and biological applications.