A Single-Loop Deep Actor-Critic Algorithm for Constrained Reinforcement Learning with Provable Convergence
Authors:
- Kexuan WangKWKexuan Wang
- An LiuALAn Liu
- Baishuo LiuBLBaishuo Liu
Preprint
Abstract
Abstract -- Deep Actor-Critic algorithms, which combine Actor-Critic with deep neural network (DNN), have been among the most prevalent reinforcement learning algorithms for decision-making problems in simulated environments. However, the existing deep Actor-Critic algorithms are still not mature to solve realistic problems with non-convex stochastic constraints and high cost to interact with the environment. In this paper, we propose a single-loop deep Actor-Critic (SLDAC) algorithmic framework for general constrained reinforcement learning (CRL) problems. In the actor step, the constrained stochastic successive convex approximation (CSSCA) method is applied to handle the non-convex stochastic objective and constraints. In the critic step, the critic DNNs are only updated once or a few finite times for each iteration, which simplifies the algorithm to a single-loop framework (the existing works require a sufficient number of updates for the critic step to ensure a good enough convergence of the inner loop for each iteration). Moreover, the variance of the policy gradient estimation is reduced by reusing observations from the old policy. The single-loop design and the observation reuse effectively reduce the agent-environment interaction cost and computational complexity. In spite of the biased policy gradient estimation incurred by the single-loop design and observation reuse, we prove that the SLDAC with a feasible initial point can converge to a Karush-Kuhn-Tuker (KKT) point of the original problem almost surely. Simulations show that the SLDAC algorithm can achieve superior performance with much lower interaction cost.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Key Technologies in 6G Terahertz Wireless Communication Systems: A SurveyIEEE Vehicular Technology Magazine, 2021
- On Finite-Time Convergence of Actor-Critic AlgorithmIEEE Journal on Selected Areas in Information Theory, 2021
- Gradient descent optimizes over-parameterized deep ReLU networksMachine Learning, 2019
- Deep Reinforcement Learning: A Brief SurveyIEEE Signal Processing Magazine, 2017
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- A Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy GradientsIEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews), 2012
- A Vector-Perturbation Technique for Near-Capacity Multiantenna Multiuser Communication—Part I: Channel Inversion and RegularizationIEEE Transactions on Communications, 2005
- Average cost temporal-difference learningAutomatica, 1999
- Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning1992
- Feasible direction methods for stochastic programming problemsMathematical Programming, 1980