Enhancing Post-Hoc Attributions in Long Document Comprehension via Coarse Grained Answer Decomposition
Preprint
Abstract
Accurately attributing answer text to its source document is crucial for developing a reliable question-answering system. However, attribution for long documents remains largely unexplored. Post-hoc attribution systems are designed to map answer text back to the source document, yet the granularity of this mapping has not been addressed. Furthermore, a critical question arises: What precisely should be attributed, with an emphasis on identifying the information units within an answer that necessitate grounding? In this paper, we propose and investigate a novel approach to the factual decomposition of generated answers for attribution, employing template-based in-context learning. To accomplish this, we utilize the question and integrate negative sampling during few-shot in-context learning for decomposition. This approach enhances the semantic understanding of both abstractive and extractive answers. We examine the impact of answer decomposition by providing a thorough examination of various attribution approaches, ranging from retrieval-based techniques to LLM-based attributors.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Improving Wikipedia verifiability with AINature Machine Intelligence, 2023
- Evaluating Attribution in Dialogue Systems: The BEGIN BenchmarkTransactions of the Association for Computational Linguistics, 2022
- A Survey on Automated Fact-CheckingTransactions of the Association for Computational Linguistics, 2022
- Large Dual Encoders Are Generalizable Retrievers2022
- Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks2022
- An Encoder Attribution Analysis for Dense Passage Retriever in Open-Domain Question Answering2022
- Towards multi-modal causability with Graph Neural Networks enabling information fusion for explainable AIInformation Fusion, 2021
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers2021
- Natural Questions: A Benchmark for Question Answering ResearchTransactions of the Association for Computational Linguistics, 2019
- Estimating the Reliability, Systematic Error and Random Error of Interval DataEducational and Psychological Measurement, 1970