Explaining the physics of transfer learning in data-driven turbulence modeling
Top Cited Papers
Open Access
- 23 January 2023
- journal article
- research article
- Published by Oxford University Press (OUP) in PNAS Nexus
- Vol. 2 (3) , pgad015
Abstract
Transfer learning (TL), which enables neural networks (NNs) to generalize out-of-distribution via targeted re-training, is becoming a powerful tool in scientific machine learning (ML) applications such as weather/climate prediction and turbulence modeling. Effective TL requires knowing (1) how to re-train NNs? and (2) what physics are learned during TL? Here, we present novel analyses and a framework addressing (1)–(2) for a broad range of multi-scale, nonlinear, dynamical systems. Our approach combines spectral (e.g. Fourier) analyses of such systems with spectral analyses of convolutional NNs, revealing physical connections between the systems and what the NN learns (a combination of low-, high-, band-pass filters and Gabor filters). Integrating these analyses, we introduce a general framework that identifies the best re-training procedure for a given problem based on physics and NN theory. As test case, we explain the physics of TL in subgrid-scale modeling of several setups of 2D turbulence. Furthermore, these analyses show that in these cases, the shallowest convolution layers are the best to re-train, which is consistent with our physics-guided framework but is against the common wisdom guiding TL in the ML literature. Our work provides a new avenue for optimal and explainable TL, and a step toward fully explainable NNs, for wide-ranging applications in science and engineering, such as climate change modeling.Keywords
Funding Information
- ONR Young Investigator Program (N00014-20-1-2722)
- NSF CSSI (OAC-2005123)
- NSF (PHY-1748958)
- NSF Xsede (ATM170020)
- NCAR (URIC0004)
This publication has 42 references indexed in Scilit:
- Deep neural networks for data-driven LES closure modelsJournal of Computational Physics, 2019
- Application of deep transfer learning for automated brain abnormality classification using MR imagesCognitive Systems Research, 2019
- Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equationsJournal of Computational Physics, 2019
- Applications of Deep Learning to Ocean Data Inference and Subgrid ParameterizationJournal of Advances in Modeling Earth Systems, 2019
- Subgrid modelling for two-dimensional turbulence using neural networksJournal of Fluid Mechanics, 2018
- Deep learning to represent subgrid processes in climate modelsProceedings of the National Academy of Sciences of the United States of America, 2018
- Solving high-dimensional partial differential equations using deep learningProceedings of the National Academy of Sciences of the United States of America, 2018
- Prognostic Validation of a Neural Network Unified Physics ParameterizationGeophysical Research Letters, 2018
- Earth System Modeling 2.0: A Blueprint for Models That Learn From Observations and Targeted High‐Resolution SimulationsGeophysical Research Letters, 2017
- Emergence of simple-cell receptive field properties by learning a sparse code for natural imagesNature, 1996