Policy Gradients Beyond Expectations: Conditional Value-at-Risk
Preprint
Abstract
Conditional Value at Risk (CVaR) is a prominent risk measure that is being used extensively in various domains. In this work we present a new formula for the gradient of the CVaR in the form of a conditional expectation. Our result is similar to policy gradients in the reinforcement learning literature. Based on this formula, we propose a novel sampling-based estimator for the CVaR gradient, and a corresponding gradient descent procedure for CVaR optimization. We analyze the bias of the estimator, and prove the convergence of the policy gradient algorithm to a local optimum. In addition, we evaluate our approach in learning a risk-sensitive controller for the game of Tetris, and propose an importance sampling procedure that is suitable for reinforcement learning.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Reinforcement learning of motor skills with policy gradientsNeural Networks, 2008
- Optimization of Convex Risk FunctionsMathematics of Operations Research, 2006
- Time Consistent Dynamic Risk MeasuresMathematical Methods of Operations Research, 2006
- Nonparametric Estimation and Sensitivity Analysis of Expected ShortfallMathematical Finance, 2003
- Risks and Portfolio Decisions Involving Hedge FundsThe Review of Financial Studies, 2003
- Stochastic Approximation and Recursive Algorithms and Applications2003
- Spectral measures of risk: A coherent representation of subjective risk aversionJournal of Banking & Finance, 2002
- A sensitivity formula for risk-sensitive cost and the actor–critic algorithmSystems & Control Letters, 2001
- Simulation-based optimization of Markov reward processesIEEE Transactions on Automatic Control, 2001
- Likelihood ratio gradient estimation for stochastic systemsCommunications of the ACM, 1990