Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior
Preprint
Abstract
Pre-trained machine learning (ML) models have shown great performance for a wide range of applications, in particular in natural language processing (NLP) and computer vision (CV). Here, we study how pre-training could be used for scientific machine learning (SciML) applications, specifically in the context of transfer learning. We study the transfer behavior of these models as (i) the pre-trained model size is scaled, (ii) the downstream training dataset size is scaled, (iii) the physics parameters are systematically pushed out of distribution, and (iv) how a single model pre-trained on a mixture of different physics problems can be adapted to various downstream applications. We find that-when fine-tuned appropriately-transfer learning can help reach desired accuracy levels with orders of magnitude fewer downstream examples (across different tasks that can even be out-of-distribution) than training from scratch, with consistent behavior across a wide range of downstream examples. We also find that fine-tuning these models yields more performance gains as model size increases, compared to training from scratch on new downstream tasks. These results hold for a broad range of PDE learning tasks. All in all, our results demonstrate the potential of the "pre-train and fine-tune" paradigm for SciML problems, demonstrating a path towards building SciML foundation models. We open-source our code for reproducibility.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- Reliable extrapolation of deep neural operators informed by physics or sparse observationsComputer Methods in Applied Mechanics and Engineering, 2023
- Residual-based adaptivity for two-phase flow simulation in porous media using Physics-informed Neural NetworksComputer Methods in Applied Mechanics and Engineering, 2022
- U-FNO—An enhanced Fourier neural operator-based deep-learning model for multiphase flowAdvances in Water Resources, 2022
- Analysis of three-dimensional potential problems in non-homogeneous media with physics-informed deep collocation method using material transfer learning and sensitivity analysisEngineering with Computers, 2022
- A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR dataComputer Methods in Applied Mechanics and Engineering, 2022
- Mosaic flows: A transferable deep learning framework for solving PDEs on unseen domainsComputer Methods in Applied Mechanics and Engineering, 2022
- Learning Elliptic Partial Differential Equations with Randomized Linear AlgebraFoundations of Computational Mathematics, 2022
- A physics-informed deep learning framework for inversion and surrogate modeling in solid mechanicsComputer Methods in Applied Mechanics and Engineering, 2021
- Transfer learning based multi-fidelity physics informed deep neural networkJournal of Computational Physics, 2021
- Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equationsJournal of Computational Physics, 2019