Guiding Autonomous Agents to Better Behaviors through Human Advice
Authors:
- Gautam KunapuliGKGautam Kunapuli
- Phillip OdomPOPhillip Odom
- Jude W. ShavlikJSJude W. Shavlik
- Sriraam NatarajanSNSriraam Natarajan
- 1 December 2013
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15504786,p. 409-418
Abstract
Inverse Reinforcement Learning (IRL) is an approach for domain-reward discovery from demonstration, where an agent mines the reward function of a Markov decision process by observing an expert acting in the domain. In the standard setting, it is assumed that the expert acts (nearly) optimally, and a large number of trajectories, i.e., training examples are available for reward discovery (and consequently, learning domain behavior). These are not practical assumptions: trajectories are often noisy, and there can be a paucity of examples. Our novel approach incorporates advice-giving into the IRL framework to address these issues. Inspired by preference elicitation, a domain expert provides advice on states and actions (features) by stating preferences over them. We evaluate our approach on several domains and show that with small amounts of targeted preference advice, learning is possible from noisy demonstrations, and requires far fewer trajectories compared to simply learning from trajectories alone.Keywords
This publication has 10 references indexed in Scilit:
- Multi-Agent Inverse Reinforcement Learning2010
- Learning to search: Functional gradient techniques for imitation learningAutonomous Robots, 2009
- A survey of robot learning from demonstrationRobotics and Autonomous Systems, 2009
- Training parsers by inverse reinforcement learningMachine Learning, 2009
- Risk-Sensitive Reinforcement Learning Applied to Control under ConstraintsJournal of Artificial Intelligence Research, 2005
- Explanation-based manipulator learning: Acquisition of planning ability through observation2005
- Apprenticeship learning via inverse reinforcement learning2004
- Modelling coaching practice: the role of instruction and demonstrationJournal of Sports Sciences, 2002
- Programming by example (introduction)Communications of the ACM, 2000
- Learning action strategies for planning domainsArtificial Intelligence, 1999