Reinforcement Learn-Based Dialogue Policy With LLM-Assisted Decision Distillation

Authors:
Open Access
Publisher website
Google Scholar
Abstract
The dialogue policy is a fundamental component of pipeline dialogue systems, responsible for steering dialogue generation and ensuring compliance with established output constraints. Traditional approaches to dialogue policy learning typically involve sequential decision-making models based on deep reinforcement learning (RL). However, these deep RL-based dialogue policies often encounter challenges related to low sample efficiency, which can hinder their performance. Recent advancements in large language models (LLMs) have demonstrated exceptional capabilities across various domains, prompting an exploration of their potential to enhance dialogue policies. In this paper, we propose a novel framework called P3L, which synergistically integrates the fine-tuning of LLMs with on-policy RL methods to optimize dialogue policy learning. Our approach employs a robust feedback mechanism to collect and analyze interaction data, facilitating the fine-tuning of LLMs. Allowing the insights gained from LLMs to be incorporated as regularization terms within the RL-based agent’s learning process. This integration of LLM decision-making aims to address the limitations of traditional RL methods by leveraging the rich contextual understanding and generative capabilities of LLMs. The experimental results demonstrate that P3L significantly enhances the efficiency of dialogue policy learning, outperforming agents that rely solely on RL techniques.
Funding Information
  • Guangdong Provincial Key Laboratory of Construction Foundation (2018B030322016)
  • Key discipline research capacity improvement project of Guangdong Province (2024ZDJS022)
  • Guangdong Provincial Key Laboratory of Construction Foundation (2026A1515011548)

This publication has 22 references indexed in Scilit: