Physics-informed Dyna-style model-based deep reinforcement learning for dynamic control
Authors:
Open Access
- 17 November 2021
- journal article
- research article
- Published by The Royal Society in Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences
- Vol. 477 (2255)
Abstract
Model-based reinforcement learning (MBRL) is believed to have much higher sample efficiency compared with model-free algorithms by learning a predictive model of the environment. However, the performance of MBRL highly relies on the quality of the learned model, which is usually built in a black-box manner and may have poor predictive accuracy outside of the data distribution. The deficiencies of the learned model may prevent the policy from being fully optimized. Although some uncertainty analysis-based remedies have been proposed to alleviate this issue, model bias still poses a great challenge for MBRL. In this work, we propose to leverage the prior knowledge of underlying physics of the environment, where the governing laws are (partially) known. In particular, we developed a physics-informed MBRL framework, where governing equations and physical constraints are used to inform the model learning and policy search. By incorporating the prior information of the environment, the quality of the learned model can be notably improved, while the required interactions with the environment are significantly reduced, leading to better sample efficiency and learning performance. The effectiveness and merit have been demonstrated over a handful of classic control problems, where the environments are governed by canonical ordinary/partial differential equations.Keywords
All Related Versions
Funding Information
- Division of Civil, Mechanical and Manufacturing Innovation (CMMI-1934300)
- Office of Cyberinfrastructure (OAC-2047127)
This publication has 43 references indexed in Scilit:
- Mastering the game of Go without human knowledgeNature, 2017
- Propagation of Input Uncertainty in Presence of Model-Form Uncertainty: A Multifidelity Approach for Computational Fluid Dynamics ApplicationsASCE-ASME Journal of Risk and Uncertainty in Engineering Systems, Part B. Mechanical Engineering, 2017
- Control of a Quadrotor With Reinforcement LearningIEEE Robotics and Automation Letters, 2017
- Physics-informed machine learning approach for reconstructing Reynolds stress modeling discrepancies based on DNS dataPhysical Review Fluids, 2017
- Reynolds averaged turbulence modelling using deep neural networks with embedded invarianceJournal of Fluid Mechanics, 2016
- Minimizing finite sums with the stochastic average gradientMathematical Programming, 2016
- Synthesis and stabilization of complex behaviors through online trajectory optimization2012
- On the State Space Geometry of the Kuramoto–Sivashinsky Flow in a Periodic DomainSIAM Journal on Applied Dynamical Systems, 2010
- Q-learningMachine Learning, 1992
- Simple statistical gradient-following algorithms for connectionist reinforcement learningMachine Learning, 1992