Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning
Preprint
Abstract
Instability and variability of Deep Reinforcement Learning (DRL) algorithms tend to adversely affect their performance. Averaged-DQN is a simple extension to the DQN algorithm, based on averaging previously learned Q-values estimates, which leads to a more stable training procedure and improved performance by reducing approximation error variance in the target values. To understand the effect of the algorithm, we examine the source of value function estimation errors and provide an analytical comparison within a simplified model. We further present experiments on the Arcade Learning Environment benchmark that demonstrate significantly improved stability and performance due to the proposed extension.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Deep Reinforcement Learning with Double Q-LearningProceedings of the AAAI Conference on Artificial Intelligence, 2016
- The Arcade Learning Environment: An Evaluation Platform for General AgentsJournal of Artificial Intelligence Research, 2013
- Learning Rates for Q-LearningLecture Notes in Computer Science, 2001
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Temporal difference learning and TD-GammonCommunications of the ACM, 1995
- Advantage Updating1993