Hindsight Credit Assignment
Authors:
- Anna HarutyunyanAHAnna Harutyunyan
- Will DabneyWDWill Dabney
- Thomas MesnardTMThomas Mesnard
- Mohammad AzarMAMohammad Azar
- Bilal PiotBPBilal Piot
- Nicolas HeessNHNicolas Heess
- Hado van HasseltHHHado van Hasselt
- Greg WayneGWGreg Wayne
- Satinder SinghSSSatinder Singh
- Doina PrecupDPDoina Precup
- Remi MunosRMRemi Munos
Preprint
Abstract
We consider the problem of efficient credit assignment in reinforcement learning. In order to efficiently and meaningfully utilize new data, we propose to explicitly assign credit to past decisions based on the likelihood of them having led to the observed outcome. This approach uses new information in hindsight, rather than employing foresight. Somewhat surprisingly, we show that value functions can be rewritten through this lens, yielding a new family of algorithms. We study the properties of these algorithms, and empirically show that they successfully address important credit assignment challenges, through a set of illustrative tasks.Keywords
All Related Versions
This publication has 2 references indexed in Scilit:
- Optimizing agent behavior over long time scales by transporting valueNature Communications, 2019
- Markov Decision Processes: Discrete Stochastic Dynamic ProgrammingTechnometrics, 1995