Learning When to Retrieve, What to Rewrite, and How to Respond in Conversational QA
Preprint
Abstract
Augmenting Large Language Models (LLMs) with information retrieval capabilities (i.e., Retrieval-Augmented Generation (RAG)) has proven beneficial for knowledge-intensive tasks. However, understanding users' contextual search intent when generating responses is an understudied topic for conversational question answering (QA). This conversational extension leads to additional concerns when compared to single-turn QA as it is more challenging for systems to comprehend conversational context and manage retrieved passages over multiple turns. In this work, we propose a method for enabling LLMs to decide when to retrieve in RAG settings given a conversational context. When retrieval is deemed necessary, the LLM then rewrites the conversation for passage retrieval and judges the relevance of returned passages before response generation. Operationally, we build on the single-turn SELF-RAG framework (Asai et al., 2023) and propose SELF-multi-RAG for conversational settings. SELF-multi-RAG demonstrates improved capabilities over single-turn variants with respect to retrieving relevant passages (by using summarized conversational context) and assessing the quality of generated responses. Experiments on three conversational QA datasets validate the enhanced response generation capabilities of SELF-multi-RAG, with improvements of ~13% measured by human annotation.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues2024
- Enhancing Chat Language Models by Scaling High-quality Instructional Conversations2023
- Detrimental Contexts in Open-Domain Question Answering2023
- Enabling Large Language Models to Generate Text with Citations2023
- Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting2023
- ConvGQR: Generative Query Reformulation for Conversational Search2023
- TopiOCQA: Open-domain Conversational Question Answering with Topic SwitchingTransactions of the Association for Computational Linguistics, 2022
- Towards a Unified Multi-Dimensional Evaluator for Text Generation2022
- CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning2022
- Open-Domain Question Answering Goes Conversational via Question Rewriting2021