Single-Loop Deep Actor-Critic for Constrained Reinforcement Learning With Provable Convergence
- 16 September 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Signal Processing
- Vol. 72 (1053587X) , 4871-4887
Abstract
Deep actor-critic (DAC) algorithms, which combine actor-critic with deep neural network (DNN), have been among the most prevalent reinforcement learning algorithms for decision-making problems in simulated environments. However, the existing DAC algorithms are still not mature to solve realistic problems with non-convex stochastic constraints and high cost to interact with the environment. In this paper, we propose a single-loop DAC (SLDAC) algorithmic framework for general constrained reinforcement learning problems. In the actor module, the constrained stochastic successive convex approximation (CSSCA) method is applied to better handle the non-convex stochastic objective and constraints. In the critic module, the critic DNNs are only updated once or a few finite times for each iteration, which simplifies the algorithm to a single-loop framework. Moreover, the variance of the policy gradient estimation is reduced by reusing observations from the old policy. The single-loop design and the observation reuse effectively reduce the agent-environment interaction cost and computational complexity. Despite the biased policy gradient estimation incurred by the single-loop design and observation reuse, we prove that the SLDAC with a feasible initial point can converge to a Karush-Kuhn-Tuker (KKT) point of the original problem almost surely. Simulations show that the SLDAC algorithm can achieve superior performance with much lower interaction cost.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62071416)
- Zhejiang Provincial Key Laboratory of Information Processing, Communication and Networking (IPCAN), Hangzhou, China (310027)
This publication has 18 references indexed in Scilit:
- Online Multi-User Scheduling for XR Transmissions With Hard-Latency Constraint: Performance Analysis and Practical DesignIEEE Transactions on Communications, 2024
- Key Technologies in 6G Terahertz Wireless Communication Systems: A SurveyIEEE Vehicular Technology Magazine, 2021
- On Finite-Time Convergence of Actor-Critic AlgorithmIEEE Journal on Selected Areas in Information Theory, 2021
- Cautious Reinforcement Learning via Distributional Risk in the Dual DomainIEEE Journal on Selected Areas in Information Theory, 2021
- Two-Stage Stochastic Optimization Via Primal-Dual Decomposition and Deep UnrollingIEEE Transactions on Signal Processing, 2021
- Resource Allocation for Secure URLLC in Mission-Critical IoT ScenariosIEEE Transactions on Communications, 2020
- Stochastic Successive Convex Approximation for Non-Convex Constrained Stochastic OptimizationIEEE Transactions on Signal Processing, 2019
- Deep Reinforcement Learning: A Brief SurveyIEEE Signal Processing Magazine, 2017
- A Vector-Perturbation Technique for Near-Capacity Multiantenna Multiuser Communication—Part I: Channel Inversion and RegularizationIEEE Transactions on Communications, 2005
- Feasible direction methods for stochastic programming problemsMathematical Programming, 1980