An Optimal Control-Based Distributed Reinforcement Learning Framework for a Class of Non-Convex Objective Functionals of the Multi-Agent Network
Authors:
- 3 November 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/CAA Journal of Automatica Sinica
- Vol. 10 (11) , 2081-2093
Abstract
This paper studies a novel distributed optimization problem that aims to minimize the sum of the non-convex objective functionals of the multi-agent network under privacy protection, which means that the local objective of each agent is unknown to others. The above problem involves complexity simultaneously in the time and space aspects. Yet existing works about distributed optimization mainly consider privacy protection in the space aspect where the decision variable is a vector with finite dimensions. In contrast, when the time aspect is considered in this paper, the decision variable is a continuous function concerning time. Hence, the minimization of the overall functional belongs to the calculus of variations. Traditional works usually aim to seek the optimal decision function. Due to privacy protection and non-convexity, the Euler-Lagrange equation of the proposed problem is a complicated partial differential equation. Hence, we seek the optimal decision derivative function rather than the decision function. This manner can be regarded as seeking the control input for an optimal control problem, for which we propose a centralized reinforcement learning (RL) framework. In the space aspect, we further present a distributed reinforcement learning framework to deal with the impact of privacy protection. Finally, rigorous theoretical analysis and simulation validate the effectiveness of our framework.Keywords
Funding Information
- National Natural Science Foundation of China (61773260)
- Ministry of Science and Technology (2018YFB 130590)
This publication has 37 references indexed in Scilit:
- Distributed Online Optimization in Dynamic Environments Using Mirror DescentIEEE Transactions on Automatic Control, 2017
- An Extremum-Seeking Controller for Distributed Optimization Over Sensor NetworksIEEE Transactions on Automatic Control, 2016
- Online Distributed Convex Optimization on Dynamic NetworksIEEE Transactions on Automatic Control, 2016
- Value and Policy Iterations in Optimal Control and Adaptive Dynamic ProgrammingIEEE Transactions on Neural Networks and Learning Systems, 2015
- Distributed gradient algorithm for constrained optimization with application to load sharing in power systemsSystems & Control Letters, 2015
- Distributed convex optimization via continuous-time coordination algorithms with discrete-time communicationAutomatica, 2015
- EXTRA: An Exact First-Order Algorithm for Decentralized Consensus OptimizationSIAM Journal on Optimization, 2015
- Consensus Based Estimation Over Relay Assisted Sensor Networks for Situation MonitoringIEEE Journal of Selected Topics in Signal Processing, 2014
- Collaborative extremum seeking for welfare optimization2014
- Online adaptive algorithm for optimal control with integral reinforcement learningInternational Journal of Robust and Nonlinear Control, 2013