Measuring and Modifying Factual Knowledge in Large Language Models
Preprint
Abstract
Large Language Models (LLMs) store an extensive amount of factual knowledge obtained from vast collections of text. To effectively utilize these models for downstream tasks, it is crucial to have reliable methods for measuring their knowledge. However, existing approaches for knowledge measurement have certain limitations, and despite recent efforts, they fail to provide accurate measurements and the necessary insights for modifying the knowledge within LLMs. In this work, we employ information theory-based measurements to provide a framework estimating the factual knowledge contained within large language models. More specifically, we measure knowledge by analyzing the LLM's prediction probability distribution before and after instilling the target knowledge, employing metrics such as entropy and KL-divergence. Introducing our metrics, we first assess their accuracy in comparison to previous ranking-based methods, surpassing them by over $35\%$ in a synthetic experiment. Then, we explore two prominent methods of knowledge instillation, discovering that LLMs exhibit limitations in capturing new knowledge under specific circumstances for one of these methods. Lastly, we demonstrate the applicability of our methods in extracting unlearned and mislearned facts in LLMs through their application to in-context learning. We make code and data for all methods and experiments in this paper publicly available.
Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- Survey of Hallucination in Natural Language GenerationACM Computing Surveys, 2023
- Crawling The Internal Knowledge-Base of Language Models2023
- Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings2022
- Retrieval Augmentation Reduces Hallucination in Conversation2021
- Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases2021