Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty
Author:
Preprint
- 1 January 2024
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
This paper studies continuous-time risk-sensitive reinforcement learning (RL) under the entropy-regularized, exploratory diffusion process formulation with the exponential-form objective. The risk-sensitive objective arises either as the agent's risk attitude or as a distributionally robust approach against the model uncertainty. Owing to the martingale perspective in Jia and Zhou (2023), the risk-sensitive RL problem is shown to be equivalent to ensuring the martingale property of a process involving both the value function and the q-function, augmented by an additional penalty term: the quadratic variation of the value process, capturing the variability of the value-to-go along the trajectory. This characterization allows for the straightforward adaptation of existing RL algorithms developed for non-risk-sensitive scenarios to incorporate risk sensitivity by adding the realized variance of the value process. Additionally, I highlight that the conventional policy gradient representation is inadequate for risk-sensitive problems due to the nonlinear nature of quadratic variation; however, q-learning offers a solution and extends to infinite horizon settings. Finally, I prove the convergence of the proposed algorithm for Merton's investment problem and quantify the impact of temperature parameter on the behavior of the learning procedure. I also conduct simulation experiments to demonstrate how risk-sensitive RL improves the finite-sample performance in the linear-quadratic control problem.Keywords
All Related Versions
This publication has 42 references indexed in Scilit:
- Robustness and ambiguity in continuous timeJournal of Economic Theory, 2011
- The exact law of large numbers via Fubini extension and characterization of insurable risksJournal of Economic Theory, 2006
- Remarks on Risk-Sensitive Control ProblemsApplied Mathematics & Optimization, 2005
- Robust control and recursive utilityFinance and Stochastics, 2003
- Robust Properties of Risk-Sensitive ControlMathematics of Control, Signals, and Systems, 2000
- Risk-Sensitive Dynamic Asset ManagementApplied Mathematics & Optimization, 1999
- Stochastic Controls1999
- Maxmin expected utility with non-unique priorJournal of Mathematical Economics, 1989
- On stochastic relaxed control for partially observed diffusionsNagoya Mathematical Journal, 1984
- Asymptotic evaluation of certain markov process expectations for large time. IVCommunications on Pure and Applied Mathematics, 1983