Risk-sensitive REINFORCE: A Monte Carlo Policy Gradient Algorithm for Exponential Performance Criteria
Authors:
- 14 December 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1522-1527
Abstract
Risk is an inherent component of any decision making process under uncertain conditions, and failure to consider risk may lead to significant performance degradation. We present a policy gradient theorem for the Risk-sensitive Control "exponential of integral" criteria, and propose a risk-sensitive Monte Carlo policy gradient algorithm. Our simulations, together with our theoretical analysis, show that the use of the exponential criteria with an appropriately chosen risk parameter not only results in a risk-sensitive policy, but also reduces variance during learning process and accelerates learning, which in turn results in a policy with higher expected return— that is to say, risk-sensitiveness leads to sample efficiency and improved performance.Keywords
This publication has 12 references indexed in Scilit:
- Natural actor–critic algorithmsAutomatica, 2009
- OnActor-Critic AlgorithmsSIAM Journal on Control and Optimization, 2003
- Robust control of set-valued discrete-time dynamical systemsIEEE Transactions on Automatic Control, 1998
- Partially Observed Differential Games, Infinite-Dimensional Hamilton–Jacobi–Isaacs Equations, and Nonlinear $H_\infty $ ControlSIAM Journal on Control and Optimization, 1996
- Robust H/sub ∞/ output feedback control for nonlinear systemsIEEE Transactions on Automatic Control, 1995
- Risk-sensitive control and dynamic games for partially observed discrete-time nonlinear systemsIEEE Transactions on Automatic Control, 1994
- Advances in prospect theory: Cumulative representation of uncertaintyJournal of Risk and Uncertainty, 1992
- Simple statistical gradient-following algorithms for connectionist reinforcement learningMachine Learning, 1992
- Optimization of stochastic linear systems with additive measurement and process noise using exponential performance criteriaIEEE Transactions on Automatic Control, 1974
- Optimal stochastic linear systems with exponential performance criteria and their relation to deterministic differential gamesIEEE Transactions on Automatic Control, 1973