PoLLMgraph: Unraveling Hallucinations in Large Language Models via State Transition Dynamics
Preprint
Abstract
Despite tremendous advancements in large language models (LLMs) over recent years, a notably urgent challenge for their practical deployment is the phenomenon of hallucination, where the model fabricates facts and produces non-factual statements. In response, we propose PoLLMgraph, a Polygraph for LLMs, as an effective model-based white-box detection and forecasting approach. PoLLMgraph distinctly differs from the large body of existing research that concentrates on addressing such challenges through black-box evaluations. In particular, we demonstrate that hallucination can be effectively detected by analyzing the LLM's internal state transition dynamics during generation via tractable probabilistic models. Experimental results on various open-source LLMs confirm the efficacy of PoLLMgraph, outperforming state-of-the-art methods by a considerable margin, evidenced by over 20% improvement in AUC-ROC on common benchmarking datasets like TruthfulQA. Our work paves a new way for model-based white-box analysis of LLMs, motivating the research community to further explore, understand, and refine the intricate dynamics of LLM behaviors.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- 🧜Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language ModelsComputational Linguistics, 2025
- TruthfulQA: Measuring How Models Mimic Human Falsehoods2022
- On Hallucination and Predictive Uncertainty in Conditional Language Generation2021
- Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics2021
- FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization2020
- A Structured Review of the Validity of BLEUComputational Linguistics, 2018
- Principal component analysisWIREs Computational Statistics, 2010
- Extraction of rules from discrete-time recurrent neural networksNeural Networks, 1996
- A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov ChainsThe Annals of Mathematical Statistics, 1970
- Error bounds for convolutional codes and an asymptotically optimum decoding algorithmIEEE Transactions on Information Theory, 1967