Adam Michael Johansen

Motivated by a recent method for approximate solution of Fredholm equations of the first kind, we develop a corresponding method for a class of Fredholm equations of the second kind. In particular, we consider the class of equations for which the solution is a probability measure. The approach centres around specifying a functional whose gradient flow admits a minimizer corresponding to a regularized version of the solution of the underlying equation and using a mean-field particle system to approximately simulate that flow. Theoretical and numerical support for the method is presented.

Summary: We present a new framework for analysing the expectation-maximization (em) algorithm. Drawing on recent advances in the theory of gradient flows over Euclidean–Wasserstein spaces, we extend techniques from alternating minimization in Euclidean spaces to the em algorithm, via its representation as coordinatewise minimization of the free energy. In so doing, we obtain finite-sample error bounds and exponential convergence of the em algorithm under a natural generalization of the log-Sobolev inequality. We further show that this framework naturally extends to several variants of the em algorithm, offering a unified approach for studying such algorithms.

We introduce a new class of sequential Monte Carlo methods which reformulates the essence of the nested sampling (NS) method of Skilling in terms of sequential Monte Carlo techniques. Two new algorithms are proposed: nested sampling via sequential Monte Carlo (NS-SMC) and adaptive nested sampling via sequential Monte Carlo (ANS-SMC). The new framework allows convergence results to be obtained in the setting when Markov chain Monte Carlo (MCMC) is used to produce new samples. An additional benefit is that marginal-likelihood (normalizing constant) estimates given by NS-SMC are unbiased. In contrast to NS, the analysis of our proposed algorithms does not require the (unrealistic) assumption that the simulated samples be independent. We show that a minor adjustment to our ANS-SMC algorithm recovers the original NS algorithm, which provides insights as to why NS seems to produce accurate estimates despite a typical violation of its assumptions. A numerical study is conducted where the performance of the proposed algorithms and temperature-annealed SMC is compared on challenging problems. Code for the experiments is made available online at https://github.com/LeahPrice/SMC-NS.

In Bayesian inverse problems, one aims at characterizing the posterior distribution of a set of unknowns, given indirect measurements. For non-linear/non-Gaussian problems, analytic solutions are seldom available: Sequential Monte Carlo samplers offer a powerful tool for approximating complex posteriors, by constructing an auxiliary sequence of densities that smoothly reaches the posterior. Often the posterior depends on a scalar hyper-parameter, for which limited prior information is available. In this work, we show that properly designed Sequential Monte Carlo (SMC) samplers naturally provide an approximation of the marginal likelihood associated with this hyper-parameter for free, i.e. at a negligible additional computational cost. The proposed method proceeds by constructing the auxiliary sequence of distributions in such a way that each of them can be interpreted as a posterior distribution corresponding to a different value of the hyper-parameter. This can be exploited to perform selection of the hyper-parameter in Empirical Bayes (EB) approaches, as well as averaging across values of the hyper-parameter according to some hyper-prior distribution in Fully Bayesian (FB) approaches. For FB approaches, the proposed method has the further benefit of allowing prior sensitivity analysis at a negligible computational cost. In addition, the proposed method exploits particles at all the (relevant) iterations, thus alleviating one of the known limitations of SMC samplers, i.e. the fact that all samples at intermediate iterations are typically discarded. We show numerical results for two distinct cases where the hyper-parameter affects only the likelihood: a toy example, where an SMC sampler is used to approximate the full posterior distribution; and a brain imaging example, where a Rao-Blackwellized SMC sampler is used to approximate the posterior distribution of a subset of parameters in a conditionally linear Gaussian model.