Jian Li
Research Subjects
Jian Li has a total of 3 co-authors. Here are the top co-authors by number of shared publications:
- Haoyu Wang
Department of Electronic Engineering, Tsinghua University, China
1 shared publication - Siyuan Wang
Department of Electronic Engineering, Tsinghua University, China
1 shared publication - Wei-Qiang Zhang
Tsinghua University
1 shared publication
Latest Publications
- Open Access
- 14 July 2023
- Published by Association for Computing Machinery (ACM)
- p. 76-81
Learnable Sparsity Structured Pruning for Acoustic Pre-trained Models
Large-scale pre-trained models bring significant gains to many speech-related tasks. However, it is still challenging to use these large models when computing power of terminal equipment is limited. Pruning is an effective method to reduce memory footprint and cost calculation. The imperfect evaluation criteria of existing pruning methods and the complex fine tuning process result in a relatively high loss of accuracy. To solve these problems, we propose a structured pruning method, which introduced the upper confidence bound of importance scores to assess the potential of each component of the model more accurately. In addition, we also introduce a set of learnable pruning threshold parameters that can be learned via stochastic gradient descent, thereby reducing the hyper-parameter tuning. We apply our method to HuBERT models on automatic speech recognition (ASR) task. Our result shows that for all pruning granularity and pruning ratios, our method yields higher accuracy and speedup ratios in the inference process.When sparsity was 60%, our method performed only 0.63% down.
- 14 July 2023
- Published by Association for Computing Machinery (ACM)
- p. 76-81
Open AccessLearnable Sparsity Structured Pruning for Acoustic Pre-trained Models
Large-scale pre-trained models bring significant gains to many speech-related tasks. However, it is still challenging to use these large models when computing power of terminal equipment is limited. Pruning is an effective method to reduce memory footprint and cost calculation. The imperfect evaluation criteria of existing pruning methods and the complex fine tuning process result in a relatively high loss of accuracy. To solve these problems, we propose a structured pruning method, which introduced the upper confidence bound of importance scores to assess the potential of each component of the model more accurately. In addition, we also introduce a set of learnable pruning threshold parameters that can be learned via stochastic gradient descent, thereby reducing the hyper-parameter tuning. We apply our method to HuBERT models on automatic speech recognition (ASR) task. Our result shows that for all pruning granularity and pruning ratios, our method yields higher accuracy and speedup ratios in the inference process.When sparsity was 60%, our method performed only 0.63% down.