UniMC: A Unified Framework for Long-Term Memory Conversation via Relevance Representation Learning
Preprint
Abstract
Open-domain long-term memory conversation can establish long-term intimacy with humans, and the key is the ability to understand and memorize long-term dialogue history information. Existing works integrate multiple models for modelling through a pipeline, which ignores the coupling between different stages. In this paper, we propose a Unified framework for Long-term Memory Conversations (UniMC), which increases the connection between different stages by learning relevance representation. Specifically, we decompose the main task into three subtasks based on probability graphs: 1) conversation summarization, 2) memory retrieval, 3) memory-augmented generation. Each subtask involves learning a representation for calculating the relevance between the query and memory, which is modelled by inserting a special token at the beginning of the decoder input. The relevance representation learning strengthens the connection across subtasks through parameter sharing and joint training. Extensive experimental results show that the proposed method consistently improves over strong baselines and yields better dialogue consistency and engagingness.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- CPT: a pre-trained unbalanced transformer for both Chinese language understanding and generationScience China Information Sciences, 2024
- EVA2.0: Investigating Open-domain Chinese Dialogue Systems with Large-scale Pre-trainingMachine Intelligence Research, 2023
- XPersona: Evaluating Multilingual Personalized Chatbot2021
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering2021
- Challenges in Building Intelligent Open-domain Dialog SystemsACM Transactions on Information Systems, 2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Personalizing Dialogue Agents: I have a dog, do you have pets too?2018
- How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation2016
- Deep Reinforcement Learning for Dialogue Generation2016
- A Persona-Based Neural Conversation Model2016