Visualizing Transformers for NLP: A Brief Survey
Authors:
- 1 September 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 270-279
Abstract
The introduction of Transformer neural networks has changed the landscape of Natural Language Processing during the last three years. While models inspired by it have managed to lead the boards for a variety of tasks, some of the mechanisms through which these performances were achieved are not necessarily well-understood. Our survey is focused mostly on explaining Transformer architectures through visualizations. Since visualization enables some degree of explainability, we have examined the various Transformer facets that can be explored through visual analytics. The field is still at a nascent stage and is expected to witness dynamic growth in the near future, since the results are already interesting and promising. Currently, some of the visualizations are relatively close to their original models, whereas others are model-agnostic. The visualizations designed to explore the Transformer architectures enable some additional features, like exploration of all neuronal cells or attention maps, therefore providing an advantage for this particular task. We conclude by proposing a set of requirements for future Transformer visualization frameworks.Keywords
This publication has 36 references indexed in Scilit:
- Visual interpretability for deep learning: a surveyFrontiers of Information Technology & Electronic Engineering, 2018
- How convolutional neural networks see the world --- A survey of convolutional neural network visualization methodsMathematical Foundations of Computing, 2018
- ConceptVector: Text Visual Analytics via Interactive Lexicon Building Using Word EmbeddingIEEE Transactions on Visualization and Computer Graphics, 2017
- LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural NetworksIEEE Transactions on Visualization and Computer Graphics, 2017
- Visualizations of Deep Neural Networks in Computer Vision: A Survey2017
- "Why Should I Trust You?"2016
- Vega-Lite: A Grammar of Interactive GraphicsIEEE Transactions on Visualization and Computer Graphics, 2016
- Parallel Coordinates for Multidimensional Data Visualization: Basic ConceptsComputing in Science & Engineering, 2015
- D³ Data-Driven DocumentsIEEE Transactions on Visualization and Computer Graphics, 2011
- Self-Organizing Maps1995