Deep Learning Language Modeling Workloads: Where Time Goes on Graphics Processors
Authors:
- 1 November 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 131-142
Abstract
Language Modeling is at the core of many natural language processing tasks. We analyze two such recent models: a Gated Convolutional Network (GCN) with five layers on the Wikitext-2 dataset and a Transformer network with 24 layers on the Google Billion Word dataset. We find that when executed on modern graphics processors, 30% - 40% of the execution time is due to the final adaptive softmax layer. Analytical modeling of the computation and memory demands of the GCN shows that this behavior will persist even if the hidden state is increased - which could be needed to improve accuracy or to support a wider vocabulary. We present variations of the adaptive softmax layer that reduce execution time for the layer by 40% and that scale better with the hidden state.Keywords
This publication has 9 references indexed in Scilit:
- fairseq: A Fast, Extensible Toolkit for Sequence Modeling2019
- Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network2018
- Hierarchical Neural Story Generation2018
- Glove: Global Vectors for Word Representation2014
- LSTM neural networks for language modeling2012
- Extensions of recurrent neural network language model2011
- Long Short-Term MemoryNeural Computation, 1997
- The Penn Treebank1994
- Human Behavior and the Principle of Least Effort.George Kingsley ZipfAmerican Journal of Sociology, 1951