A Survey of Sim-to-Real Transfer Techniques Applied to Reinforcement Learning for Bioinspired Robots
- 29 September 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 34 (7) , 3444-3459
Abstract
The state-of-the-art reinforcement learning (RL) techniques have made innumerable advancements in robot control, especially in combination with deep neural networks (DNNs), known as deep reinforcement learning (DRL). In this article, instead of reviewing the theoretical studies on RL, which were almost fully completed several decades ago, we summarize some state-of-the-art techniques added to commonly used RL frameworks for robot control. We mainly review bioinspired robots (BIRs) because they can learn to locomote or produce natural behaviors similar to animals and humans. With the ultimate goal of practical applications in real world, we further narrow our review scope to techniques that could aid in sim-to-real transfer. We categorized these techniques into four groups: 1) use of accurate simulators; 2) use of kinematic and dynamic models; 3) use of hierarchical and distributed controllers; and 4) use of demonstrations. The purposes of these four groups of techniques are to supply general and accurate environments for RL training, improve sampling efficiency, divide and conquer complex motion tasks and redundant robot structures, and acquire natural skills. We found that, by synthetically using these techniques, it is possible to deploy RL on physical BIRs in actuality.Keywords
Funding Information
- Japan Society for the Promotion of Science (JSPS) Kakenhi (JP20KK0256)
This publication has 79 references indexed in Scilit:
- Robust spring mass model running for a physical bipedal robot2015
- Learning legged swimming gaits from experience2015
- Learning to Adjust and Refine Gait Patterns for a Biped RobotIEEE Transactions on Systems, Man, and Cybernetics: Systems, 2015
- Gait synthesis of a hybrid legged robot using reinforcement learning2015
- Optimized Assistive Human–Robot Interaction Using Reinforcement LearningIEEE Transactions on Cybernetics, 2015
- Human-level control through deep reinforcement learningNature, 2015
- Motion capture and reinforcement learning of dynamically stable humanoid movement primitives2013
- DCOB: Action space for reinforcement learning of high DoF robotsAutonomous Robots, 2013
- Real-world reinforcement learning for autonomous humanoid robot dockingRobotics and Autonomous Systems, 2012
- Learning variable impedance controlThe International Journal of Robotics Research, 2011