TPrune
Authors:
Open Access
- 15 April 2021
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Cyber-Physical Systems
- Vol. 5 (3) , 1-22
Abstract
The invention of Transformer model structure boosts the performance of Neural Machine Translation (NMT) tasks to an unprecedented level. Many previous works have been done to make the Transformer model more execution-friendly on resource-constrained platforms. These researches can be categorized into three key fields: Model Pruning, Transfer Learning, and Efficient Transformer Variants. The family of model pruning methods are popular for their simplicity in practice and promising compression rate and have achieved great success in the field of convolution neural networks (CNNs) for many vision tasks. Nonetheless, previous Transformer pruning works did not perform a thorough model analysis and evaluation on each Transformer component on off-the-shelf mobile devices. In this work, we analyze and prune transformer models at the line-wise granularity and also implement our pruning method on real mobile platforms. We explore the properties of all Transformer components as well as their sparsity features, which are leveraged to guide Transformer model pruning. We name our whole Transformer analysis and pruning pipeline as TPrune. In TPrune, we first propose Block-wise Structured Sparsity Learning (BSSL) to analyze Transformer model property. Then, based on the characters derived from BSSL, we apply Structured Hoyer Square (SHS) to derive the final pruned models. Comparing with the state-of-the-art Transformer pruning methods, TPrune is able to achieve a higher model compression rate with less performance degradation. Experimental results show that our pruned models achieve 1.16×–1.92× speedup on mobile devices with 0%–8% BLEU score degradation compared with the original Transformer model.Keywords
Funding Information
- National Science Foundation (CNS-1717657, CCF-1937435, CNS-1822085)
This publication has 11 references indexed in Scilit:
- MobiEye2019
- ZARA2019
- HyPar: Towards Hybrid Parallelism for Deep Learning Accelerator Array2019
- NeuralHMC2019
- SPN dash2018
- Training Tips for the Transformer ModelThe Prague Bulletin of Mathematical Linguistics, 2018
- Running sparse and low-precision neural network: When algorithm meets hardware2018
- MeDNN: A distributed mobile system with enhanced partition and deployment for large-scale DNNs2017
- MoDNN: Local distributed mobile computing system for Deep Neural Network2017
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998