Deep Reinforcement Learning with Averaged Target DQN
Authors:
Preprint
Abstract
The commonly used Q-learning algorithm combined with function approximation induces systematic overestimations of state-action values. These systematic errors might cause instability, poor performance and sometimes divergence of learning. In this work, we present the Averaged Target DQN (ADQN) algorithm, an adaptation to the DQN class of algorithms which uses a weighted average over past learned networks to reduce generalization noise variance. As a consequence, this leads to reduced overestimations, more stable learning process and improved performance. Additionally, we analyze ADQN variance reduction along trajectories and demonstrate the performance of ADQN on a toy Gridworld problem, as well as on several of the Atari 2600 games from the Arcade Learning Environment.Keywords
All Related Versions
This publication has 2 references indexed in Scilit:
- Deep Reinforcement Learning with Double Q-LearningProceedings of the AAAI Conference on Artificial Intelligence, 2016
- The Arcade Learning Environment: An Evaluation Platform for General AgentsJournal of Artificial Intelligence Research, 2013