Deep Reinforcement Learning with Averaged Target DQN
Authors:
Preprint
Abstract
The commonly used Q-learning algorithm combined with function approximation induces systematic overestimations of state-action values. These systematic errors might cause instability, poor performance and sometimes divergence of learning. In this work, we present the \textsc{Averaged Target DQN} (ADQN) algorithm, an adaptation to the DQN class of algorithms which uses a weighted average over past learned networks to reduce generalization noise variance. As a consequence, this leads to reduced overestimations, more stable learning process and improved performance. Additionally, we analyze ADQN variance reduction along trajectories and demonstrate the performance of ADQN on a toy Gridworld problem, as well as on several of the Atari 2600 games from the Arcade Learning Environment.Keywords
All Related Versions
This publication has 2 references indexed in Scilit:
- Deep Reinforcement Learning with Double Q-LearningProceedings of the AAAI Conference on Artificial Intelligence, 2016
- The Arcade Learning Environment: An Evaluation Platform for General AgentsJournal of Artificial Intelligence Research, 2013