Augmenting LLMs with Knowledge: A survey on hallucination prevention
Preprint
Abstract
Large pre-trained language models have demonstrated their proficiency in storing factual knowledge within their parameters and achieving remarkable results when fine-tuned for downstream natural language processing tasks. Nonetheless, their capacity to access and manipulate knowledge with precision remains constrained, resulting in performance disparities on knowledge-intensive tasks when compared to task-specific architectures. Additionally, the challenges of providing provenance for model decisions and maintaining up-to-date world knowledge persist as open research frontiers. To address these limitations, the integration of pre-trained models with differentiable access mechanisms to explicit non-parametric memory emerges as a promising solution. This survey delves into the realm of language models (LMs) augmented with the ability to tap into external knowledge sources, including external knowledge bases and search engines. While adhering to the standard objective of predicting missing tokens, these augmented LMs leverage diverse, possibly non-parametric external modules to augment their contextual processing capabilities, departing from the conventional language modeling paradigm. Through an exploration of current advancements in augmenting large language models with knowledge, this work concludes that this emerging research direction holds the potential to address prevalent issues in traditional LMs, such as hallucinations, un-grounded responses, and scalability challenges.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion2022
- Fine-tuned Language Models are Continual Learners2022
- Internet-Augmented Dialogue Generation2022
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering2021
- Recipes for Building an Open-Domain Chatbot2021
- Dense Passage Retrieval for Open-Domain Question Answering2020
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- Billion-Scale Similarity Search with GPUsIEEE Transactions on Big Data, 2019
- PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text2019
- Topic-sensitive PageRank2002