Reward-rational (implicit) choice: A unifying formalism for reward learning
Preprint
Abstract
It is often difficult to hand-specify what the correct reward function is for a task, so researchers have instead aimed to learn reward functions from human behavior or feedback. The types of behavior interpreted as evidence of the reward function have expanded greatly in recent years. We've gone from demonstrations, to comparisons, to reading into the information leaked when the human is pushing the robot away or turning it off. And surely, there is more to come. How will a robot make sense of all these diverse types of behavior? Our key insight is that different types of behavior can be interpreted in a single unifying formalism - as a reward-rational choice that the human is making, often implicitly. The formalism offers both a unifying lens with which to view past work, as well as a recipe for interpreting new sources of information that are yet to be uncovered. We provide two examples to showcase this: interpreting a new feedback type, and reading into how the choice of feedback itself leaks information about the reward.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Simplifying Reward Design through Divide-and-Conquer2018
- Unified Pragmatic Models for Generating and Following Instructions2018
- Learning preferences for manipulation tasks from online coactive feedbackThe International Journal of Robotics Research, 2015
- Motion planning with sequential convex optimization and convex collision checkingThe International Journal of Robotics Research, 2014
- Thermodynamics as a theory of decision-making with information-processing costsProceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2013
- Action understanding as inverse planningCognition, 2009
- Impedance Control: An Approach to Manipulation: Part II—ImplementationJournal of Dynamic Systems, Measurement, and Control, 1985
- The Analysis of PermutationsJournal of the Royal Statistical Society Series C: Applied Statistics, 1975