Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learning
Preprint
Abstract
Building a dialogue agent to fulfill complex tasks, such as travel planning, is challenging because the agent has to learn to collectively complete multiple subtasks. For example, the agent needs to reserve a hotel and book a flight so that there leaves enough time for commute between arrival and hotel check-in. This paper addresses this challenge by formulating the task in the mathematical framework of options over Markov Decision Processes (MDPs), and proposing a hierarchical deep reinforcement learning approach to learning a dialogue manager that operates at different temporal scales. The dialogue manager consists of: (1) a top-level dialogue policy that selects among subtasks or options, (2) a low-level dialogue policy that selects primitive actions to complete the subtask given by the top-level policy, and (3) a global state tracker that helps ensure all cross-subtask constraints be satisfied. Experiments on a travel planning task with simulated and real users show that our approach leads to significant improvements over three baselines, two based on handcrafted rules and the other based on flat deep reinforcement learning.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Scaling up deep reinforcement learning for multi-domain dialogue systems2017
- Frames: a corpus for adding memory to goal-oriented dialogue systems2017
- A Sequence-to-Sequence Model for User Simulation in Spoken Dialogue Systems2016
- Spoken language understanding using long short-term memory neural networks2014
- POMDP-Based Statistical Spoken Dialog Systems: A ReviewProceedings of the IEEE, 2013
- The Hidden Agenda User Simulation ModelIEEE Transactions on Audio, Speech, and Language Processing, 2009
- Recent Advances in Hierarchical Reinforcement LearningDiscrete Event Dynamic Systems, 2003
- A stochastic model of human-machine interaction for learning dialog strategiesIEEE Transactions on Speech and Audio Processing, 2000
- Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learningArtificial Intelligence, 1999