Enhancing Contextual Understanding in Large Language Models through Contrastive Decoding
Preprint
Abstract
Large language models (LLMs) tend to inadequately integrate input context during text generation, relying excessively on encoded prior knowledge in model parameters, potentially resulting in generated text with factual inconsistencies or contextually unfaithful content. LLMs utilize two primary knowledge sources: 1) prior (parametric) knowledge from pretraining, and 2) contextual (non-parametric) knowledge from input prompts. The study addresses the open question of how LLMs effectively balance these knowledge sources during the generation process, specifically in the context of open-domain question answering. To address this issue, we introduce a novel approach integrating contrastive decoding with adversarial irrelevant passages as negative samples to enhance robust context grounding during generation. Notably, our method operates at inference time without requiring further training. We conduct comprehensive experiments to demonstrate its applicability and effectiveness, providing empirical evidence showcasing its superiority over existing methodologies. Our code is publicly available at: https://github.com/amazon-science/ContextualUnderstanding-ContrastiveDecoding.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Survey of Hallucination in Natural Language GenerationACM Computing Surveys, 2023
- How Can We Know When Language Models Know? On the Calibration of Language Models for Question AnsweringTransactions of the Association for Computational Linguistics, 2021
- Entity-Based Knowledge Conflicts in Question Answering2021
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering2021
- DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts2021
- Reducing Quantity Hallucinations in Abstractive Summarization2020
- How Much Knowledge Can You Pack Into the Parameters of a Language Model?2020
- On Faithfulness and Factuality in Abstractive Summarization2020
- Natural Questions: A Benchmark for Question Answering ResearchTransactions of the Association for Computational Linguistics, 2019
- The Probabilistic Relevance Framework: BM25 and BeyondFoundations and Trends® in Information Retrieval, 2009