Generalization in Monitored Markov Decision Processes (Mon-MDPs)
Authors:
Preprint
Abstract
Reinforcement learning (RL) typically models the interaction between the agent and environment as a Markov decision process (MDP), where the rewards that guide the agent's behavior are always observable. However, in many real-world scenarios, rewards are not always observable, which can be modeled as a monitored Markov decision process (Mon-MDP). Prior work on Mon-MDPs have been limited to simple, tabular cases, restricting their applicability to real-world problems. This work explores Mon-MDPs using function approximation (FA) and investigates the challenges involved. We show that combining function approximation with a learned reward model enables agents to generalize from monitored states with observable rewards, to unmonitored environment states with unobservable rewards. Therefore, we demonstrate that such generalization with a reward model achieves near-optimal policies in environments formally defined as unsolvable. However, we identify a critical limitation of such function approximation, where agents incorrectly extrapolate rewards due to overgeneralization, resulting in undesirable behaviors. To mitigate overgeneralization, we propose a cautious police optimization method leveraging reward uncertainty. This work serves as a step towards bridging this gap between Mon-MDP theory and real-world applications.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Achieving Human Level Competitive Robot Table Tennis2025
- Loss of plasticity in deep continual learningNature, 2024
- GVFs in the real world: making predictions online for water treatmentMachine Learning, 2023
- Outracing champion Gran Turismo drivers with deep reinforcement learningNature, 2022
- Autonomous navigation of stratospheric balloons using reinforcement learningNature, 2020
- Grandmaster level in StarCraft II using multi-agent reinforcement learningNature, 2019
- DeepStack: Expert-level artificial intelligence in heads-up no-limit pokerScience, 2017
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- Optimal control of Markov processes with incomplete state informationJournal of Mathematical Analysis and Applications, 1965
- DYNAMIC PROGRAMMING AND STATISTICAL COMMUNICATION THEORYProceedings of the National Academy of Sciences, 1957