Does DQN Learn?
- 10 November 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automatic Control
- Vol. 71 (4) , 2482-2495
Abstract
A primary requirement for any reinforcement learning method is that it should produce policies that improve upon the initial guess. In this work, we show that the widely used Deep Q-Network (DQN) fails to satisfy this minimal criterion—even when it gets to see all possible states and actions infinitely often (a condition under which tabular Q-learning is guaranteed to converge to the optimal Q-value function). Our specific contributions are twofold. First, we numerically show that DQN often returns a policy that performs worse than the initial one. Second, we offer a theoretical explanation for this phenomenon in linear DQN, a simplified version of DQN that uses linear function approximation in place of neural networks while retaining the other key components such as $\epsilon$-greedy exploration, experience replay, and target network. Using tools from differential inclusion theory, we prove that the limit points of linear DQN correspond to fixed points of projected Bellman operators. Crucially, we show that these fixed points need not relate to optimal—or even near-optimal—policies, thus explaining linear DQN's sub-optimal behaviors. We also give a scenario where linear DQN always identifies the worst policy. Our work fills a longstanding gap in understanding the convergence behaviors of Q-learning with function approximation and $\epsilon$-greedy exploration.Keywords
This publication has 20 references indexed in Scilit:
- Stochastic Approximation: A Dynamical Systems Viewpoint2022
- Full Gradient DQN Reinforcement Learning: A Provably Convergent Scheme2021
- Multiscale Q-learning with linear function approximationDiscrete Event Dynamic Systems, 2015
- Approximate policy iteration: a survey and some new methodsControl Theory and Technology, 2011
- Neuro-Dynamic Programming2001
- Open Theoretical Questions in Reinforcement Learning1999
- Dynamics of stochastic approximation algorithms1999
- Residual Algorithms: Reinforcement Learning with Function Approximation1995
- Differential Equations with Discontinuous Righthand Sides1988
- Differential Inclusions1984