Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shifts
Preprint
Abstract
We study the robustness of deep reinforcement learning algorithms against distribution shifts within contextual multi-stage stochastic combinatorial optimization problems from the operations research domain. In this context, risk-sensitive algorithms promise to learn robust policies. While this field is of general interest to the reinforcement learning community, most studies up-to-date focus on theoretical results rather than real-world performance. With this work, we aim to bridge this gap by formally deriving a novel risk-sensitive deep reinforcement learning algorithm while providing numerical evidence for its efficacy. Specifically, we introduce discrete Soft Actor-Critic for the entropic risk measure by deriving a version of the Bellman equation for the respective Q-values. We establish a corresponding policy improvement result and infer a practical algorithm. We introduce an environment that represents typical contextual multi-stage stochastic combinatorial optimization problems and perform numerical experiments to empirically validate our algorithm's robustness against realistic distribution shifts, without compromising performance on the training distribution. We show that our algorithm is superior to risk-neutral Soft Actor-Critic as well as to two benchmark approaches for robust deep reinforcement learning. Thereby, we provide the first structured analysis on the robustness of reinforcement learning under distribution shifts in the realm of contextual multi-stage stochastic combinatorial optimization problems.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Risk-Sensitive Reinforcement Learning With Exponential CriteriaIEEE Transactions on Cybernetics, 2025
- Graph Soft Actor–Critic Reinforcement Learning for Large-Scale Distributed Multirobot CoordinationIEEE Transactions on Neural Networks and Learning Systems, 2023
- High robustness energy management strategy of hybrid electric vehicle based on improved soft actor-critic deep reinforcement learningEnergy, 2022
- A Deep Reinforcement Learning Algorithm Suitable for Autonomous Vehicles: Double Bootstrapped Soft-Actor–Critic-DiscreteIEEE Transactions on Cognitive and Developmental Systems, 2021
- Motion Planning for Dual-Arm Robot Based on Soft Actor-CriticIEEE Access, 2021
- Use of Proximal Policy Optimization for the Joint Replenishment ProblemComputers in Industry, 2020
- Learning to Walk Via Deep Reinforcement Learning2019
- Variance-constrained actor-critic algorithms for discounted and average reward MDPsMachine Learning, 2016
- Smooth minimization of non-smooth functionsMathematical Programming, 2004
- Coherent Measures of RiskMathematical Finance, 1999