Luke Dicks

Prior beliefs about the latent function to shape inductive biases can be incorporated into a Gaussian process (GP) via the kernel. However, beyond kernel choices, the decision-making process of GP models remains poorly understood. In this work, we contribute an analysis of the loss landscape for GP models using methods from chemical physics. We demonstrate ν-continuity for Matérn kernels and outline aspects of catastrophe theory at critical points in the loss landscape. By directly including ν in the hyperparameter optimisation for Matérn kernels, we find that typical values of ν can be far from optimal in terms of performance. We also provide an a priori method for evaluating the effect of GP ensembles and discuss various voting approaches based on physical properties of the loss landscape. The utility of these approaches is demonstrated for various synthetic and real datasets. Our findings provide insight into hyperparameter optimisation for GPs and offer practical guidance for improving their performance and interpretability in a range of applications.

By drawing on similarities between energy landscapes and model response surfaces we gain new insight into model performance, even in the absence of data.

In this work, we outline how methods from the energy landscapes field of theoretical chemistry can be applied to study machine learning models. Various applications are found, ranging from interpretability to improved model performance.

The story of machine learning in general, and its application to molecular design in particular, has been a tale of evolving representations of data. Understanding the implications of the use of a particular representation -- including the existence of so-called `activity cliffs' for cheminformatics models -- is the key to their successful use for molecular discovery. In this work we present a physics-inspired methodology which exploits analogies between model response surfaces and energy landscapes to richly describe the relationship between the representation and the model. From these similarities, a metric emerges which is analogous to the commonly used frustration metric from the chemical physics community. This new property shows state-of-the-art prediction of model error, whilst belonging to a novel class of roughness measure that extends beyond the known data allowing the trivial identification of activity cliffs even in the absence of related training or evaluation data.

Rotamers, namely amino acid side chain conformations common to many different peptides, can be compiled into libraries. These rotamer libraries are used in protein modeling, where the limited conformational space occupied by amino acid side chains is exploited. Here, we construct a sequence-dependent rotamer library from simulations of all possible tripeptides, which provides rotameric states dependent on adjacent amino acids. We observe significant sensitivity of rotamer populations to sequence and find that the library is successful in locating side chain conformations present in crystal structures. The library is designed for applications with basin-hopping global optimization, where we use it to propose moves in conformational space. The addition of rotamer moves significantly increases the efficiency of protein structure prediction within this framework, and we determine parameters to optimize efficiency.

The K-means algorithm, routinely used in many scientific fields, generates clustering solutions that depend on the initial cluster coordinates. The number of solutions may be large, which can make locating the global minimum challenging. Hence, the topography of the cost function surface is crucial to understanding the performance of the algorithm. Here, we employ the energy landscape approach to elucidate the topography of the K-means cost function surface for Fisher’s Iris dataset. For any number of clusters, we find that the solution landscapes have a funneled structure that is usually associated with efficient global optimization. An analysis of the barriers between clustering solutions shows that the funneled structures result from remarkably small barriers between almost all clustering solutions. The funneled structure becomes less well-defined as the number of clusters increases, and we analyze kinetic analogs to quantify the increased difficulty in locating the global minimum for these different landscapes.