Model-free Control Design Using Policy Gradient Reinforcement Learning in LPV Framework

Authors:
  • YBYajie Bao
    Yajie Bao
  • JVJavad Mohammadpour Velni
    Javad Mohammadpour Velni

Abstract
This paper presents an off-policy policy gradient reinforcement learning (RL) approach to control nonlinear systems in linear parameter-varying (LPV) framework. The (parameter-varying) controller is learned from the closed-loop trajectories using off-policy actor-critic methods for RL, instead of being designed based on the system model. To decrease constraint violation (and hence improve safety), exploration around a valid control sequence is proposed to facilitate learning before applying the real policy. Additionally, the valid control sequence is used to determine the complexity of the actor and critic neural networks (NN). The uncertainties in the evolution of LPV scheduling variables are tackled by generating episodes with varying scheduling trajectories. Adapting to unseen scheduling signals is empirically investigated and fine-tuning is used to refine the learned controller. Furthermore, the prior knowledge of the control law (e.g., static state feedback) is embedded in designing the structure of actor NN. Experiments using two numerical examples and a control moment gyroscope (CMG) simulation model illustrate the success of the proposed approach for stabilization and tracking control in the presence of uncertainty in LPV scheduling variables.
Funding Information
  • National Science Foundation