Efficient Epistemic Uncertainty Estimation for Large Language Models via Knowledge Distillation
Authors:
- Seonghyeon ParkSPSeonghyeon Park
- Jewon YeomJYJewon Yeom
- Jaewon SokJSJaewon Sok
- Jeongjae ParkJPJeongjae Park
- Heejun KimHKHeejun Kim
- Taesup KimTKTaesup Kim
Preprint
Abstract
Quantifying uncertainty in Large Language Models (LLMs) is essential for mitigating hallucinations and enabling risk-aware deployment in safety-critical tasks. However, estimating Epistemic Uncertainty(EU) via Deep Ensembles is computationally prohibitive at the scale of modern models. We propose a framework that leverages the small draft models to efficiently estimate token-level EU, bypassing the need for full-scale ensembling. Theoretically grounded in a Bias-Variance Decomposition, our approach approximates EU via Jensen-Shannon divergence among drafts (variance proxy) and KL divergence between the draft mixture and the target (bias proxy). To further ensure accuracy without significant overhead, we introduce Online Stochastic Distillation (OSD) to efficiently approximate target aggregation and the Data-Diverse Drafts (DDD) strategy to enhance draft diversity for better target approximation. Extensive experiments on GSM8K demonstrate that our method reduces the estimation error (RMSE) by up to 37% compared to baselines. Crucially, our approach achieves Hallucination Detection performance competitive with heavy perturbation-based methods like TokUR while incurring negligible inference costs, offering a practical solution for uncertainty-aware LLM deployment.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Large language models hallucination: A comprehensive surveyComputer Science Review, 2026
- Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and ApplicationACM Transactions on Intelligent Systems and Technology, 2025
- Calibrating Large Language Models with Sample ConsistencyProceedings of the AAAI Conference on Artificial Intelligence, 2025
- Dynamic-Width Speculative Beam Decoding for LLM InferenceProceedings of the AAAI Conference on Artificial Intelligence, 2025
- A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open QuestionsACM Transactions on Information Systems, 2025
- Detecting hallucinations in large language models using semantic entropyNature, 2024
- A survey of uncertainty in deep neural networksArtificial Intelligence Review, 2023
- Survey of Hallucination in Natural Language GenerationACM Computing Surveys, 2023
- How Can We Know When Language Models Know? On the Calibration of Language Models for Question AnsweringTransactions of the Association for Computational Linguistics, 2021
- Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methodsMachine Learning, 2021