Johannes Brandstetter

Physics simulators are essential in science and engineering, enabling the analysis, control, and design of complex systems. In experimental sciences, they are increasingly used to automate experimental design, often via combinatorial search and optimization. However, as the setups grow more complex, the computational cost of traditional, CPU-based simulators becomes a major limitation. Here, we show how neural surrogate models can significantly reduce reliance on such slow simulators while preserving accuracy. Taking the design of interferometric gravitational wave detectors as a representative example, we train a neural network to surrogate the gravitational wave physics simulator Finesse, which was developed by the LIGO community. Despite that small changes in physical parameters can change the output by orders of magnitudes, the model rapidly predicts the quality and feasibility of candidate designs, allowing an efficient exploration of large design spaces. Our algorithm loops between training the surrogate, designing new experiments, and verifying their properties with the slow simulator for further training. Assisted by auto-differentiation and GPU parallelism, our method proposes high-quality experiments much faster than direct optimization. Within hours, our algorithm’s findings outperform designs that the original optimizer takes five days to reach. Beyond detecting gravitational waves, our surrogate approach can be adapted to other interferometric systems. Moreover, our active learning approach is broadly applicable to other domains where simulator bottlenecks hinder optimization and discovery.

Driven by the advancement of GPUs and AI, the field of Computational Fluid Dynamics (CFD) is undergoing significant transformations. This paper bridges the gap between the machine learning and CFD communities by deconstructing industrial-scale CFD simulations into their core components. Our main contribution is to propose the first scaling law that incorporates CFD inputs for both data generation and model training to outline the unique challenges of developing and deploying these next-generation AI models for complex fluid dynamics problems. Using our new scaling law, we establish quantitative estimates for the large-scale limit, distinguishing between regimes where the cost of data generation is the dominant factor in total compute versus where the cost of model training prevails. We conclude that the incorporation of high-fidelity transient data provides the optimum route to a foundation model. We constrain our theory with concrete numbers, providing the first public estimates on the computational cost and time to build a foundation model for CFD.

Simulation-based plasma scenario development plays a crucial role in designing next-generation tokamaks and fusion power plants. However, the inclusion of high-fidelity simulations of Scrape-Off Layer (SOL) turbulence and transient MHD events such as Edge Localized Modes (ELMs) in highly iterative applications remains computationally prohibitive, limiting their use in design and control workflows. Understanding these phenomena is vital, as they govern heat flux on plasma-facing components, influencing reactor performance and material lifetime. This study explored Fourier Neural Operators (FNOs) as surrogate models to accelerate plasma simulations from the JOREK MHD and STORM turbulence codes. FNOs were trained on single-step rollouts and evaluated in terms of long-term predictive accuracy in an auto-regressive manner. To mitigate the computational burden of dataset generation, a transfer learning strategy was explored, leveraging low-fidelity simulations to improve performance on high-fidelity datasets. These results showed that FNOs effectively captured initial plasma evolution, including blob movement and density source localization for JOREK and STORM, respectively. However, long rollouts accumulated errors and exhibited sensitivity to certain physical phenomena, leading to non-monotonic error spikes. Transfer learning significantly reduced errors for small dataset sizes and short rollouts, achieving an order-of-magnitude reduction when transfering from low- to high-fidelity datasets. However, its effectiveness diminished with longer rollouts and larger dataset sizes, especially when applied to datasets with significantly different dynamics. Attempts to transfer models to previously unseen variables in simulations were unsuccessful, underscoring the limitations of transfer learning in this context. These findings demonstrate the promise of neural operators for accelerating fusion-relevant PDE simulations. However, they also highlight key challenges: improving long-term accuracy to mitigate error accumulation, capturing critical physical behaviors, and developing robust surrogates that effectively leverage multi-fidelity, multi-physics datasets.

A central mechanism in machine learning is to identify, store, and recognize patterns. How to learn, access, and retrieve such patterns is crucial in Hopfield networks and the more recent transformer architectures. We show that the attention mechanism of transformer architectures is actually the update rule of modern Hop-field networks that can store exponentially many patterns. We exploit this high storage capacity of modern Hopfield networks to solve a challenging multiple instance learning (MIL) problem in computational biology: immune repertoire classification. Accurate and interpretable machine learning methods solving this problem could pave the way towards new vaccines and therapies, which is currently a very relevant research topic intensified by the COVID-19 crisis. Immune repertoire classification based on the vast number of immunosequences of an individual is a MIL problem with an unprecedentedly massive number of instances, two orders of magnitude larger than currently considered problems, and with an extremely low witness rate. In this work, we present our novel method DeepRC that integrates transformer-like attention, or equivalently modern Hopfield networks, into deep learning architectures for massive MIL such as immune repertoire classification. We demonstrate that DeepRC outperforms all other methods with respect to predictive performance on large-scale experiments, including simulated and real-world virus infection data, and enables the extraction of sequence motifs that are connected to a given disease class. Source code and datasets: https://github.com/ml-jku/DeepRC