Influential Training Data Retrieval for Explaining Verbalized Confidence of LLMs
Preprint
Abstract
Large language models (LLMs) can increase users' perceived trust by verbalizing confidence in their outputs. However, prior work has shown that LLMs are often overconfident, making their stated confidence unreliable since it does not consistently align with factual accuracy. To better understand the sources of this verbalized confidence, we introduce TracVC (\textbf{Trac}ing \textbf{V}erbalized \textbf{C}onfidence), a method that builds on information retrieval and influence estimation to trace generated confidence expressions back to the training data. We evaluate TracVC on OLMo and Llama models in a question answering setting, proposing a new metric, content groundness, which measures the extent to which an LLM grounds its confidence in content-related training examples (relevant to the question and answer) versus in generic examples of confidence verbalization. Our analysis reveals that OLMo2-13B is frequently influenced by confidence-related data that is lexically unrelated to the query, suggesting that it may mimic superficial linguistic expressions of certainty rather than rely on genuine content grounding. These findings point to a fundamental limitation in current training regimes: LLMs may learn how to sound confident without learning when confidence is justified. Our analysis provides a foundation for improving LLMs' trustworthiness in expressing more reliable confidence.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- Confronting verbalized uncertainty: Understanding how LLM’s verbalized uncertainty influences users in AI-assisted decision-makingInternational Journal of Human-Computer Studies, 2025
- Are Large Language Models More Honest in Their Probabilistic or Verbalized Confidence?2025
- What large language models know and what people think they knowNature Machine Intelligence, 2025
- Training data influence analysis and estimation: a surveyMachine Learning, 2024
- Identifying a Training-Set Attack's Target Using Renormalized Influence Estimation2022
- Scaling Up Influence FunctionsProceedings of the AAAI Conference on Artificial Intelligence, 2022
- TruthfulQA: Measuring How Models Mimic Human Falsehoods2022
- FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging2021
- Natural Questions: A Benchmark for Question Answering ResearchTransactions of the Association for Computational Linguistics, 2019
- Multivariate Density Estimation1992