Pretraining in Deep Reinforcement Learning: A Survey
Authors:
- Zichuan LinZLZichuan Lin
- Junyou LiJLJunyou Li
- Shuai LiSLShuai Li
- Deheng YeDYDeheng Ye
Preprint
Abstract
The past few years have seen rapid progress in combining reinforcement learning (RL) with deep learning. Various breakthroughs ranging from games to robotics have spurred the interest in designing sophisticated RL algorithms and systems. However, the prevailing workflow in RL is to learn tabula rasa, which may incur computational inefficiency. This precludes continuous deployment of RL algorithms and potentially excludes researchers without large-scale computing resources. In many other areas of machine learning, the pretraining paradigm has shown to be effective in acquiring transferable knowledge, which can be utilized for a variety of downstream tasks. Recently, we saw a surge of interest in Pretraining for Deep RL with promising results. However, much of the research has been based on different experimental settings. Due to the nature of RL, pretraining in this field is faced with unique challenges and hence requires new design principles. In this survey, we seek to systematically review existing works in pretraining for deep reinforcement learning, provide a taxonomy of these methods, discuss each sub-field, and bring attention to open problems and future directions.Keywords
All Related Versions
This publication has 35 references indexed in Scilit:
- Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretraining2022
- Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms2021
- Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methodsMachine Learning, 2021
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- A survey of robot learning from demonstrationRobotics and Autonomous Systems, 2009
- An analysis of model-based Interval Estimation for Markov Decision ProcessesJournal of Computer and System Sciences, 2008
- Equivalence notions and model minimization in Markov decision processesArtificial Intelligence, 2003
- Recent Advances in Hierarchical Reinforcement LearningDiscrete Event Dynamic Systems, 2003
- Bisimulation through probabilistic testingInformation and Computation, 1991
- An algorithm for quadratic programmingNaval Research Logistics Quarterly, 1956