Explainability for Large Language Models: A Survey
Preprint
Abstract
Large language models (LLMs) have demonstrated impressive capabilities in natural language processing. However, their internal mechanisms are still unclear and this lack of transparency poses unwanted risks for downstream applications. Therefore, understanding and explaining these models is crucial for elucidating their behaviors, limitations, and social impacts. In this paper, we introduce a taxonomy of explainability techniques and provide a structured overview of methods for explaining Transformer-based language models. We categorize techniques based on the training paradigms of LLMs: traditional fine-tuning-based paradigm and prompting-based paradigm. For each paradigm, we summarize the goals and dominant approaches for generating local explanations of individual predictions and global explanations of overall model knowledge. We also discuss metrics for evaluating generated explanations, and discuss how explanations can be leveraged to debug models and improve performance. Lastly, we examine key challenges and emerging opportunities for explanation techniques in the era of LLMs in comparison to conventional machine learning models.Keywords
All Related Versions
This publication has 80 references indexed in Scilit:
- Look Before You Leap: An Exploratory Study of Uncertainty Analysis for Large Language ModelsIEEE Transactions on Software Engineering, 2025
- AttentionViz: A Global View of Transformer AttentionIEEE Transactions on Visualization and Computer Graphics, 2023
- Algorithms to estimate Shapley value feature attributionsNature Machine Intelligence, 2023
- VisQA: X-raying Vision and Language Reasoning in TransformersIEEE Transactions on Visualization and Computer Graphics, 2021
- Transformer Interpretability Beyond Attention Visualization2021
- Attention Flows: Analyzing and Comparing Attention Mechanisms in Language ModelsIEEE Transactions on Visualization and Computer Graphics, 2020
- SANVis: Visual Analytics for Understanding Self-Attention Networks2019
- On Attribution of Recurrent Neural Network Predictions via Additive Decomposition2019
- Seq2seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence ModelsIEEE Transactions on Visualization and Computer Graphics, 2018
- Explaining nonlinear classification decisions with deep Taylor decompositionPattern Recognition, 2017