RL + Model-Based Control: Using On-Demand Optimal Control to Learn Versatile Legged Locomotion
- 21 August 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 8 (10) , 6619-6626
Abstract
This letter presents a control framework that combines model-based optimal control and reinforcement learning (RL) to achieve versatile and robust legged locomotion. Our approach enhances the RL training process by incorporating on-demand reference motions generated through finite-horizon optimal control, covering a broad range of velocities and gaits. These reference motions serve as targets for the RL policy to imitate, leading to the development of robust control policies that can be learned with reliability. Furthermore, by utilizing realistic simulation data that captures whole-body dynamics, RL effectively overcomes the inherent limitations in reference motions imposed by modeling simplifications. We validate the robustness and controllability of the RL training process within our framework through a series of experiments. In these experiments, our method showcases its capability to generalize reference motions and effectively handle more complex locomotion tasks that may pose challenges for the simplified model, thanks to RL's flexibility. Additionally, our framework effortlessly supports the training of control policies for robots with diverse dimensions, eliminating the necessity for robot-specific adjustments in the reward function and hyperparameters.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Motion Planning for Quadrupedal Locomotion: Coupled Planning, Terrain Mapping, and Whole-Body ControlIEEE Transactions on Robotics, 2020
- A Computational Framework for Designing Skilled Legged-Wheeled RobotsIEEE Robotics and Automation Letters, 2020
- Online Relative Footstep Optimization for Legged Robots Dynamic Walking Using Discrete-Time Model Predictive Control2019
- Mini Cheetah: A Platform for Pushing the Limits of Dynamic Quadruped Control2019
- Real-time Model Predictive Control for Versatile Dynamic Motions in Quadrupedal Robots2019
- Learning agile and dynamic motor skills for legged robotsScience Robotics, 2019
- Dynamic Locomotion in the MIT Cheetah 3 Through Convex Model-Predictive Control2018
- DeepMimicACM Transactions on Graphics, 2018
- Gait and Trajectory Optimization for Legged Systems Through Phase-Based End-Effector ParameterizationIEEE Robotics and Automation Letters, 2018
- Policy-regularized model predictive control to stabilize diverse quadrupedal gaits for the MIT cheetah2017