Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
Authors:
Top Cited Papers
- 1 February 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 464-483
Abstract
The last decade of machine learning has seen drastic increases in scale and capabilities. Deep neural networks (DNNs) are increasingly being deployed in the real world. However, they are difficult to analyze, raising concerns about using them without a rigorous understanding of how they function. Effective tools for interpreting them will be important for building more trustworthy AI by helping to identify problems, fix bugs, and improve basic understanding. In particular, “inner” interpretability techniques, which focus on explaining the internal components of DNNs, are well-suited for developing a mechanistic understanding, guiding manual modifications, and reverse engineering solutions. Much recent work has focused on DNN interpretability, and rapid progress has thus far made a thorough systematization of methods difficult. In this survey, we review over 300 works with a focus on inner interpretability tools. We introduce a taxonomy that classifies methods by what part of the network they help to explain (weights, neurons, subnetworks, or latent representations) and whether they are implemented during (intrinsic) or after (post hoc) training. To our knowledge, we are also the first to survey a number of connections between interpretability research and work in adversarial robustness, continual learning, modularity, network compression, and studying the human visual system. We discuss key challenges and argue that the status quo in interpretability research is largely unproductive. Finally, we highlight the importance of future work that emphasizes diagnostics, debugging, adversaries, and benchmarking in order to make interpretability tools more useful to engineers in practical applications.Keywords
All Related Versions
This publication has 110 references indexed in Scilit:
- Modular representation of layered neural networksNeural Networks, 2018
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences, 2017
- Generating Visual Explanations2016
- Future Progress in Artificial Intelligence: A Survey of Expert Opinion2016
- Neural Module Networks2016
- Probing for semantic evidence of composition by means of simple classification tasks2016
- Understanding deep image representations by inverting them2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Distributional vectors encode referential attributes2015
- Representation Learning: A Review and New PerspectivesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013