Understanding Retrieval Augmentation for Long-Form Question Answering
Preprint
Abstract
We present a study of retrieval-augmented language models (LMs) on long-form question answering. We analyze how retrieval augmentation impacts different LMs, by comparing answers generated from models while using the same evidence documents, and how differing quality of retrieval document set impacts the answers generated from the same LM. We study various attributes of generated answers (e.g., fluency, length, variance) with an emphasis on the attribution of generated long-form answers to in-context evidence documents. We collect human annotations of answer attribution and evaluate methods for automatically judging attribution. Our study provides new insights on how retrieval augmentation impacts long, knowledge-rich text generation of LMs. We further identify attribution patterns for long text generation and analyze the main culprits of attribution errors. Together, our analysis reveals how retrieval augmentation impacts long knowledge-rich text generation and provide directions for future work.Keywords
All Related Versions
This publication has 29 references indexed in Scilit:
- Survey of Hallucination in Natural Language GenerationACM Computing Surveys, 2023
- SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in SummarizationTransactions of the Association for Computational Linguistics, 2022
- TRUE: Re-evaluating Factual Consistency Evaluation2022
- Entity-Based Knowledge Conflicts in Question Answering2021
- SimCSE: Simple Contrastive Learning of Sentence Embeddings2021
- DocNLI: A Large-scale Dataset for Document-level Natural Language Inference2021
- Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence2021
- Hurdles to Progress in Long-form Question Answering2021
- Adversarial NLI: A New Benchmark for Natural Language Understanding2020
- The Probabilistic Relevance Framework: BM25 and BeyondFoundations and Trends® in Information Retrieval, 2009