Deep Q-Learning With Gradient Target Tracking
Open Access
- 3 February 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 14, 29648-29673
Abstract
This paper introduces Q-learning with gradient target tracking, a novel reinforcement learning framework that provides a learned continuous target update mechanism as an alternative to the conventional hard update paradigm. In the standard deep Q-network (DQN), the target network is a copy of the online network’s weights, held fixed for a number of iterations before being periodically replaced via a hard update. While this stabilizes training by providing consistent targets, it introduces a new challenge: the hard update period must be carefully tuned to achieve optimal performance. To address this issue, we propose two gradient-based target update methods: DQN with asymmetric gradient target tracking (AGT2-DQN) and DQN with symmetric gradient target tracking (SGT2-DQN). These methods replace the conventional hard target updates with continuous and structured updates using gradient descent, which effectively eliminates the need for manual tuning. We provide a theoretical analysis proving the convergence of these methods in tabular settings. Additionally, empirical evaluations demonstrate their advantages over standard DQN baselines, which suggest that gradient-based target updates can serve as an effective alternative to conventional target update mechanisms in Q-learning.Keywords
All Related Versions
Funding Information
- Institute of Information Communications Technology Planning Evaluation
- Korean government (2022-0-00469)
- Brain Korea (BK) 21 Fostering Outstanding Universities for Research (FOUR) from the Ministry of Education
This publication has 16 references indexed in Scilit:
- Finite-Time Analysis of Asynchronous Q-Learning Under Diminishing Step-Size From Control-Theoretic ViewIEEE Access, 2024
- Human-level control through deep reinforcement learningNature, 2015
- Stochastic Recursive Algorithms for Optimization2013
- Error bounds for constant step-size -learningSystems & Control Letters, 2012
- Stability and Stabilizability of Switched Linear Systems: A Survey of Recent ResultsIEEE Transactions on Automatic Control, 2009
- Chapter 4 Monotone Dynamical Systems2006
- Switching in Systems and Control2003
- Ordinary Differential Equations1998
- Learning to predict by the methods of temporal differencesMachine Learning, 1988
- Equilibrium points in n -person gamesProceedings of the National Academy of Sciences of the United States of America, 1950