Donghwan Lee
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Donghwan Lee has a total of 55 co-authors. Here are the top co-authors by number of shared publications:
- Do Wan Kim
Department of Electrical Engineering, Hanbat National University, Daejeon, South Korea
12 shared publications - Jianghai Hu
Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA
7 shared publications - Niao He
Department of Computer Science, ETH Zurich, Zurich, Switzerland
6 shared publications Jianghai Hu
Purdue University
4 shared publicationsHan-Dong Lim
3 shared publications- Pedro Luis Dias Peres
Faculdade de Engenharia Eltrica e de Computao, Universidade Estadual de Campinas (UNICAMP), Campinas, Brazil
3 shared publications
Latest Publications
- Open Access
- 11 August 2026
- Published by Springer Nature in International Journal of Control, Automation, and Systems
- Vol. 24 (9) , 2181-2195
TD-Learning and Q-Learning: A Survey of Theory, Analysis, and Trends
This paper provides a comprehensive survey of the convergence properties of temporal-difference (TD) learning and Q-learning, which are two fundamental algorithms in reinforcement learning (RL). We systematically categorize the existing literature into the tabular setting and two function approximation regimes: linear and nonlinear (deep neural networks). In the tabular setting, we review the foundational stochastic-approximation theory that ensures asymptotic convergence and discuss recent non-asymptotic results that provide explicit sample-complexity bounds under various coverage and sampling conditions. For linear function approximation, we address the stability challenges inherent in off-policy learning and examine stabilization mechanisms, such as gradient-based methods, regularization, and target networks. Furthermore, we explore the recent theoretical advancements in deep RL, focusing on finite-time analysis within the overparameterized regime. By synthesizing these diverse perspectives, this survey highlights the theoretical evolution from asymptotic stability to non-asymptotic efficiency and identifies the remaining gaps and provides a coherent roadmap for future research toward a unified theoretical understanding of RL dynamics.
- 11 August 2026
- Published by Springer Nature in International Journal of Control, Automation, and Systems
- Vol. 24 (9) , 2181-2195
Open AccessTD-Learning and Q-Learning: A Survey of Theory, Analysis, and Trends
This paper provides a comprehensive survey of the convergence properties of temporal-difference (TD) learning and Q-learning, which are two fundamental algorithms in reinforcement learning (RL). We systematically categorize the existing literature into the tabular setting and two function approximation regimes: linear and nonlinear (deep neural networks). In the tabular setting, we review the foundational stochastic-approximation theory that ensures asymptotic convergence and discuss recent non-asymptotic results that provide explicit sample-complexity bounds under various coverage and sampling conditions. For linear function approximation, we address the stability challenges inherent in off-policy learning and examine stabilization mechanisms, such as gradient-based methods, regularization, and target networks. Furthermore, we explore the recent theoretical advancements in deep RL, focusing on finite-time analysis within the overparameterized regime. By synthesizing these diverse perspectives, this survey highlights the theoretical evolution from asymptotic stability to non-asymptotic efficiency and identifies the remaining gaps and provides a coherent roadmap for future research toward a unified theoretical understanding of RL dynamics.
- Open Access
- 27 May 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Systems, Man, and Cybernetics: Systems
- Vol. PP (21682216) , 1-15
Regional Output-Feedback Control for T–S Fuzzy Systems Using Lyapunov Functions With Polynomial Dependency on States and Membership Functions
This article addresses the problem of local output-feedback control design for continuous-time Takagi–Sugeno fuzzy systems using the parallel distributed compensation (PDC) approach. The main contribution is the introduction of a novel class of Lyapunov functions with polynomial dependence on both the system states and the membership functions (MFs), providing less conservative stability and synthesis conditions. To compute the PDC gains, a locally convergent iterative algorithm based on linear matrix inequalities (LMIs) is proposed. Unlike many existing methods, the approach eliminates the need for bounds on the time derivatives of MFs by explicitly incorporating their dynamics into the stability conditions. The algorithm operates in two phases: first, optimizing the decay rate of the closed-loop trajectories, andsecond, maximizing the estimated domain of attraction (DOA) under a fixed decay rate. Comparative numerical examples based on benchmark systems demonstrate the superior performance of the proposed method over existing techniques, particularly as the polynomial degree of the Lyapunov function increases.
- 27 May 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Systems, Man, and Cybernetics: Systems
- Vol. PP (21682216) , 1-15
Open AccessRegional Output-Feedback Control for T–S Fuzzy Systems Using Lyapunov Functions With Polynomial Dependency on States and Membership Functions
This article addresses the problem of local output-feedback control design for continuous-time Takagi–Sugeno fuzzy systems using the parallel distributed compensation (PDC) approach. The main contribution is the introduction of a novel class of Lyapunov functions with polynomial dependence on both the system states and the membership functions (MFs), providing less conservative stability and synthesis conditions. To compute the PDC gains, a locally convergent iterative algorithm based on linear matrix inequalities (LMIs) is proposed. Unlike many existing methods, the approach eliminates the need for bounds on the time derivatives of MFs by explicitly incorporating their dynamics into the stability conditions. The algorithm operates in two phases: first, optimizing the decay rate of the closed-loop trajectories, andsecond, maximizing the estimated domain of attraction (DOA) under a fixed decay rate. Comparative numerical examples based on benchmark systems demonstrate the superior performance of the proposed method over existing techniques, particularly as the polynomial degree of the Lyapunov function increases.
- Open Access
- 12 February 2026
- Published by Elsevier in SSRN Electronic Journal
Finite-Time Accuracy of Temporal-Difference Learning Under Schur-Stable Recursions
Temporal difference (TD) learning is a cornerstone reinforcement learning (RL) method for policy evaluation, where the goal is to estimate the value function of a Markov decision process under a fixed policy. While a substantial body of work has established its convergence and stability properties, more recent efforts have focused on its statistical efficiency through finite-time error bounds. In this paper, we advance this line of research by developing a new finite-time error analysis for tabular TD learning that directly exploits a discrete-time stochastic linear system representation and leverages Schur stability of the associated matrices. Beyond the specific bounds obtained, the proposed framework provides a reusable template for analyzing TD learning and related RL algorithms, and it offers control-theoretic insights that may guide future developments in finite-sample RL theory.
- 12 February 2026
- Published by Elsevier in SSRN Electronic Journal
Open AccessFinite-Time Accuracy of Temporal-Difference Learning Under Schur-Stable Recursions
Temporal difference (TD) learning is a cornerstone reinforcement learning (RL) method for policy evaluation, where the goal is to estimate the value function of a Markov decision process under a fixed policy. While a substantial body of work has established its convergence and stability properties, more recent efforts have focused on its statistical efficiency through finite-time error bounds. In this paper, we advance this line of research by developing a new finite-time error analysis for tabular TD learning that directly exploits a discrete-time stochastic linear system representation and leverages Schur stability of the associated matrices. Beyond the specific bounds obtained, the proposed framework provides a reusable template for analyzing TD learning and related RL algorithms, and it offers control-theoretic insights that may guide future developments in finite-sample RL theory.
- Open Access
- 3 February 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 14, 29648-29673
Deep Q-Learning With Gradient Target Tracking
This paper introduces Q-learning with gradient target tracking, a novel reinforcement learning framework that provides a learned continuous target update mechanism as an alternative to the conventional hard update paradigm. In the standard deep Q-network (DQN), the target network is a copy of the online network’s weights, held fixed for a number of iterations before being periodically replaced via a hard update. While this stabilizes training by providing consistent targets, it introduces a new challenge: the hard update period must be carefully tuned to achieve optimal performance. To address this issue, we propose two gradient-based target update methods: DQN with asymmetric gradient target tracking (AGT2-DQN) and DQN with symmetric gradient target tracking (SGT2-DQN). These methods replace the conventional hard target updates with continuous and structured updates using gradient descent, which effectively eliminates the need for manual tuning. We provide a theoretical analysis proving the convergence of these methods in tabular settings. Additionally, empirical evaluations demonstrate their advantages over standard DQN baselines, which suggest that gradient-based target updates can serve as an effective alternative to conventional target update mechanisms in Q-learning.
- 3 February 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 14, 29648-29673
Open AccessDeep Q-Learning With Gradient Target Tracking
This paper introduces Q-learning with gradient target tracking, a novel reinforcement learning framework that provides a learned continuous target update mechanism as an alternative to the conventional hard update paradigm. In the standard deep Q-network (DQN), the target network is a copy of the online network’s weights, held fixed for a number of iterations before being periodically replaced via a hard update. While this stabilizes training by providing consistent targets, it introduces a new challenge: the hard update period must be carefully tuned to achieve optimal performance. To address this issue, we propose two gradient-based target update methods: DQN with asymmetric gradient target tracking (AGT2-DQN) and DQN with symmetric gradient target tracking (SGT2-DQN). These methods replace the conventional hard target updates with continuous and structured updates using gradient descent, which effectively eliminates the need for manual tuning. We provide a theoretical analysis proving the convergence of these methods in tabular settings. Additionally, empirical evaluations demonstrate their advantages over standard DQN baselines, which suggest that gradient-based target updates can serve as an effective alternative to conventional target update mechanisms in Q-learning.
- 6 January 2026
- Published by Elsevier in Neurocomputing
- Vol. 671
Finite-time analysis of simultaneous double Q-learning
- 6 January 2026
- Published by Elsevier in Neurocomputing
- Vol. 671
Finite-time analysis of simultaneous double Q-learning
- Open Access
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 14, 128581-128593
High-Precision Temperature Control in Industrial Evaporation Heaters via Hybrid Reinforcement Learning Approach
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 14, 128581-128593
Open AccessHigh-Precision Temperature Control in Industrial Evaporation Heaters via Hybrid Reinforcement Learning Approach
- Open Access
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Fuzzy Systems
- Vol. 34 (3) , 843-857
Regional Stabilization and Estimation of Domains of Attraction for Discrete-Time T–S Fuzzy Systems via Fuzzy-modeled Membership Functions
This paper addresses the problem of regional stabilization for discrete-time Takagi-Sugeno fuzzy systems using parallel distributed compensation output-feedback controllers. The goal is to ensure asymptotic stability and to maximize the estimate of the corresponding domain of attraction. The key contribution of this work is the use of a fuzzy modeling approach for the membership functions, enabling a polytopic representation based on the premise variables. Unlike conventional approaches, this method avoids the need for explicit bounds on membership function variations, thereby reducing conservatism in the stabilization conditions. The synthesis procedure is presented as an iterative algorithm based on linear matrix inequalities, consisting of two distinct phases: the first phase stabilizes the system, while the second phase aims to maximize the estimated region of attraction. Numerical examples are provided to demonstrate the advantages of the proposed technique over existing methods.
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Fuzzy Systems
- Vol. 34 (3) , 843-857
Open AccessRegional Stabilization and Estimation of Domains of Attraction for Discrete-Time T–S Fuzzy Systems via Fuzzy-modeled Membership Functions
This paper addresses the problem of regional stabilization for discrete-time Takagi-Sugeno fuzzy systems using parallel distributed compensation output-feedback controllers. The goal is to ensure asymptotic stability and to maximize the estimate of the corresponding domain of attraction. The key contribution of this work is the use of a fuzzy modeling approach for the membership functions, enabling a polytopic representation based on the premise variables. Unlike conventional approaches, this method avoids the need for explicit bounds on membership function variations, thereby reducing conservatism in the stabilization conditions. The synthesis procedure is presented as an iterative algorithm based on linear matrix inequalities, consisting of two distinct phases: the first phase stabilizes the system, while the second phase aims to maximize the estimated region of attraction. Numerical examples are provided to demonstrate the advantages of the proposed technique over existing methods.
- Open Access
Adaptive Policy Backbone via Shared Network
Reinforcement learning (RL) has achieved impressive results across domains, yet learning an optimal policy typically requires extensive interaction data, limiting practical deployment. A common remedy is to leverage priors, such as pre-collected datasets or reference policies, but their utility degrades under task mismatch between training and deployment. While prior work has sought to address this mismatch, it has largely been restricted to in-distribution settings. To address this challenge, we propose Adaptive Policy Backbone (APB), a meta-transfer RL method that inserts lightweight linear layers before and after a shared backbone, thereby enabling parameter-efficient fine-tuning (PEFT) while preserving prior knowledge during adaptation. Our results show that APB improves sample efficiency over standard RL and adapts to out-of-distribution (OOD) tasks where existing meta-RL baselines typically fail.
Open AccessAdaptive Policy Backbone via Shared Network
Reinforcement learning (RL) has achieved impressive results across domains, yet learning an optimal policy typically requires extensive interaction data, limiting practical deployment. A common remedy is to leverage priors, such as pre-collected datasets or reference policies, but their utility degrades under task mismatch between training and deployment. While prior work has sought to address this mismatch, it has largely been restricted to in-distribution settings. To address this challenge, we propose Adaptive Policy Backbone (APB), a meta-transfer RL method that inserts lightweight linear layers before and after a shared backbone, thereby enabling parameter-efficient fine-tuning (PEFT) while preserving prior knowledge during adaptation. Our results show that APB improves sample efficiency over standard RL and adapts to out-of-distribution (OOD) tasks where existing meta-RL baselines typically fail.
- Open Access
- 1 September 2025
- Published by International Joint Conferences on Artificial Intelligence
- Vol. 7, 5698-5706
A Primal-dual Perspective for Distributed TD-learning
The goal of this paper is to investigate distributed temporal difference (TD) learning for a networked multi-agent Markov decision process. The proposed approach is based on distributed optimization algorithms, which can be interpreted as primal-dual ordinary differential equation (ODE) dynamics subject to null-space constraints. Based on the exponential convergence behavior of the primal-dual ODE dynamics subject to null-space constraints, we examine the behavior of the final iterate in various distributed TD-learning scenarios, considering both constant and diminishing step-sizes and incorporating both i.i.d. and Markovian observation models. Unlike existing methods, the proposed algorithm does not require the assumption that the underlying communication network structure is characterized by a doubly stochastic matrix.
- 1 September 2025
- Published by International Joint Conferences on Artificial Intelligence
- Vol. 7, 5698-5706
Open AccessA Primal-dual Perspective for Distributed TD-learning
The goal of this paper is to investigate distributed temporal difference (TD) learning for a networked multi-agent Markov decision process. The proposed approach is based on distributed optimization algorithms, which can be interpreted as primal-dual ordinary differential equation (ODE) dynamics subject to null-space constraints. Based on the exponential convergence behavior of the primal-dual ODE dynamics subject to null-space constraints, we examine the behavior of the final iterate in various distributed TD-learning scenarios, considering both constant and diminishing step-sizes and incorporating both i.i.d. and Markovian observation models. Unlike existing methods, the proposed algorithm does not require the assumption that the underlying communication network structure is characterized by a doubly stochastic matrix.
- Open Access
- 12 August 2025
- Published by Springer Nature in Research Square
Robust Deterministic Policy Gradient for Disturbance Attenuation and Its Application to Quadrotor Control
Practical control systems pose significant challenges in identifying optimal control policies due to uncertainties in the system model and external disturbances. While \(H_\infty\) control techniques are commonly used to design robust controllers that mitigate the effects of disturbances, these methods often require complex and computationally intensive calculations. To address this issue, this paper proposes a reinforcement learning algorithm called Robust Deterministic Policy Gradient (RDPG), which formulates the $H_\infty$ control problem as a two-player zero-sum dynamic game. In this formulation, one player (the user) aims to minimize the cost, while the other player (the adversary) seeks to maximize it. We then employ deterministic policy gradient (DPG) and its deep reinforcement learning counterpart to train a robust control policy with effective disturbance attenuation. In particular, for practical implementation, we introduce an algorithm called robust deep deterministic policy gradient (RDDPG), which employs a deep neural network architecture and integrates techniques from the twin-delayed deep deterministic policy gradient (TD3) to enhance stability and learning efficiency.To evaluate the proposed algorithm, we implement it on an unmanned aerial vehicle (UAV) tasked with following a predefined path in a disturbance-prone environment. The experimental results demonstrate that the proposed method outperforms other control approaches in terms of robustness against disturbances, enabling precise real-time tracking of moving targets even under severe disturbance conditions.- 12 August 2025
- Published by Springer Nature in Research Square
Open AccessRobust Deterministic Policy Gradient for Disturbance Attenuation and Its Application to Quadrotor Control
Practical control systems pose significant challenges in identifying optimal control policies due to uncertainties in the system model and external disturbances. While \(H_\infty\) control techniques are commonly used to design robust controllers that mitigate the effects of disturbances, these methods often require complex and computationally intensive calculations. To address this issue, this paper proposes a reinforcement learning algorithm called Robust Deterministic Policy Gradient (RDPG), which formulates the $H_\infty$ control problem as a two-player zero-sum dynamic game. In this formulation, one player (the user) aims to minimize the cost, while the other player (the adversary) seeks to maximize it. We then employ deterministic policy gradient (DPG) and its deep reinforcement learning counterpart to train a robust control policy with effective disturbance attenuation. In particular, for practical implementation, we introduce an algorithm called robust deep deterministic policy gradient (RDDPG), which employs a deep neural network architecture and integrates techniques from the twin-delayed deep deterministic policy gradient (TD3) to enhance stability and learning efficiency.To evaluate the proposed algorithm, we implement it on an unmanned aerial vehicle (UAV) tasked with following a predefined path in a disturbance-prone environment. The experimental results demonstrate that the proposed method outperforms other control approaches in terms of robustness against disturbances, enabling precise real-time tracking of moving targets even under severe disturbance conditions.