Introduction to optimization methods for training SciML models
Preprint
Abstract
Optimization is central to both modern machine learning (ML) and scientific machine learning (SciML), yet the structure of the underlying optimization problems differs substantially across these domains. Classical ML typically relies on stochastic, sample-separable objectives that favor first-order and adaptive gradient methods. In contrast, SciML often involves physics-informed or operator-constrained formulations in which differential operators induce global coupling, stiffness, and strong anisotropy in the loss landscape. As a result, optimization behavior in SciML is governed by the spectral properties of the underlying physical models rather than by data statistics, frequently limiting the effectiveness of standard stochastic methods and motivating deterministic or curvature-aware approaches. This document provides a unified introduction to optimization methods in ML and SciML, emphasizing how problem structure shapes algorithmic choices. We review first- and second-order optimization techniques in both deterministic and stochastic settings, discuss their adaptation to physics-constrained and data-driven SciML models, and illustrate practical strategies through tutorial examples, while highlighting open research directions at the interface of scientific computing and scientific machine learning.Keywords
All Related Versions
This publication has 44 references indexed in Scilit:
- Machine learning and domain decomposition methods - a surveyComputational Science and Engineering, 2024
- Multilevel domain decomposition-based architectures for physics-informed neural networksComputer Methods in Applied Mechanics and Engineering, 2024
- Residual-based attention in physics-informed neural networksComputer Methods in Applied Mechanics and Engineering, 2024
- Finite basis physics-informed neural networks (FBPINNs): a scalable domain decomposition approach for solving differential equationsAdvances in Computational Mathematics, 2023
- Multilevel‐in‐width training for deep neural network regressionNumerical Linear Algebra with Applications, 2023
- A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networksComputer Methods in Applied Mechanics and Engineering, 2022
- Surrogate modelling for an aircraft dynamic landing loads simulation using an LSTM AutoEncoder-based dimensionality reduction approachAerospace Science and Technology, 2022
- On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networksComputer Methods in Applied Mechanics and Engineering, 2021
- Conservative physics-informed neural networks on discrete domains for conservation laws: Applications to forward and inverse problemsComputer Methods in Applied Mechanics and Engineering, 2020
- A posteriori error estimation and adaptive mesh-refinement techniquesJournal of Computational and Applied Mathematics, 1994