Risk-Sensitive Reinforcement Learning With Exponential Criteria
- 18 June 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cybernetics
- Vol. 55 (8) , 3774-3787
Abstract
While reinforcement learning (RL) has shown experimental success in a number of applications, it is known to be sensitive to noise and perturbations in the parameters of the system, leading to high variability in the total reward amongst different episodes on slightly different environments. To introduce robustness, as well as sample efficiency, risk-sensitive RL methods are being thoroughly studied. In this work, we provide a definition of robust RL policies and formulate a risk-sensitive RL problem to approximate them, by solving an optimization problem with respect to a modified objective based on exponential criteria. In particular, we study a model-free risk-sensitive variation of the widely used Monte Carlo policy gradient algorithm, and introduce a novel risk-sensitive online Actor-Critic algorithm based on solving a multiplicative Bellman equation using stochastic approximation updates. Analytical results suggest that the use of exponential criteria generalizes commonly used ad-hoc regularization approaches, improves sample efficiency, and introduces robustness with respect to perturbations in the model parameters and the environment. The implementation, performance, and robustness properties of the proposed methods are evaluated in simulated experiments.Keywords
All Related Versions
Funding Information
- U.S. Office of Naval Research (N00014-17-1-2622)
- U.S. NSF (ECCS2127605)
- Clark Foundation Distinguished Fellowship
- University of Maryland Graduate School Ann G. Wylie Fellowship
This publication has 21 references indexed in Scilit:
- Risk-sensitive Reinforcement Learning and Robust Learning for Control2021
- Maximum-Entropy Progressive State Aggregation for Reinforcement Learning2021
- Kernel dynamic policy programming: Applicable reinforcement learning to robot systems with high dimensional statesNeural Networks, 2017
- Policy Gradients for CVaR-Constrained MDPs2014
- Stochastic approximationResonance, 2013
- Convex measures of risk and trading constraintsFinance and Stochastics, 2002
- Robust control of set-valued discrete-time dynamical systemsIEEE Transactions on Automatic Control, 1998
- Robust H/sub ∞/ output feedback control for nonlinear systemsIEEE Transactions on Automatic Control, 1995
- Risk-sensitive control and dynamic games for partially observed discrete-time nonlinear systemsIEEE Transactions on Automatic Control, 1994
- Function Optimization using Connectionist Reinforcement Learning AlgorithmsConnection Science, 1991