A framework for training larger networks for deep Reinforcement learning
Open Access
- 5 June 2024
- journal article
- research article
- Published by Springer Nature in Machine Learning
- Vol. 113 (9) , 6115-6139
Abstract
The success of deep learning in computer vision and natural language processing communities can be attributed to the training of very deep neural networks with millions or billions of parameters, which can then be trained with massive amounts of data. However, a similar trend has largely eluded the training of deep reinforcement learning (RL) algorithms where larger networks do not lead to performance improvement. Previous work has shown that this is mostly due to instability during the training of deep RL agents when using larger networks. In this paper, we make an attempt to understand and address the training of larger networks for deep RL. We first show that naively increasing network capacity does not improve performance. Then, we propose a novel method that consists of (1) wider networks with DenseNet connection, (2) decoupling representation learning from the training of RL, and (3) a distributed training method to mitigate overfitting problems. Using this three-fold technique, we show that we can train very large networks that result in significant performance gains. We present several ablation studies to demonstrate the efficacy of the proposed method and some intuitive understanding of the reasons for performance gain. We show that our proposed method outperforms other baseline algorithms on several challenging locomotion tasks.Keywords
This publication has 19 references indexed in Scilit:
- StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learning2022
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- Wider or Deeper: Revisiting the ResNet Model for Visual RecognitionPattern Recognition, 2019
- State representation learning for control: An overviewNeural Networks, 2018
- Densely Connected Convolutional Networks2017
- Learning state representation for deep actor-critic control2016
- Identity Mappings in Deep Residual Networks2016
- Deep Residual Learning for Image Recognition2016
- Human-level control through deep reinforcement learningNature, 2015
- ImageNet: A large-scale hierarchical image database2009