Haoyu Wang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Haoyu Wang has a total of 4 co-authors. Here are the top co-authors by number of shared publications:
- Wei-Qiang Zhang
Tsinghua University
3 shared publications - Siyuan Wang
Department of Electronic Engineering, Tsinghua University, China
2 shared publications - Jian Li
Sinovoice, China
1 shared publication - Yaguang Gong
TAL Education, China
1 shared publication
Latest Publications
- 25 July 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Signal Processing
- Vol. 18 (6) , 1046-1058
Unstructured Pruning and Low Rank Factorisation of Self-Supervised Pre-Trained Speech Models
Self-supervised pre-trained speech models require significant memory and computational resources, limiting their applicability to many speech tasks. Unstructured pruning is a compression method that can achieve minimal performance degradation, while the resulting sparse matrix mandates special hardware or computational operators for acceleration. In this study, we propose a novel approach that leverages the potential low-rank structures of the unstructured sparse matrices by applying truncated singular value decomposition (SVD), thus converting them into parameter-efficient dense models. Moreover, we introduce nuclear norm regularisation to ensure lower rank and a learnable singular value selection strategy to determine the approximate truncation rate for each matrix. Experiments on multiple speech tasks demonstrate that the proposed method can convert an unstructured sparse model into a light-weight and hardware-friendly dense model with comparable or superior performance.
- 25 July 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Signal Processing
- Vol. 18 (6) , 1046-1058
Unstructured Pruning and Low Rank Factorisation of Self-Supervised Pre-Trained Speech Models
Self-supervised pre-trained speech models require significant memory and computational resources, limiting their applicability to many speech tasks. Unstructured pruning is a compression method that can achieve minimal performance degradation, while the resulting sparse matrix mandates special hardware or computational operators for acceleration. In this study, we propose a novel approach that leverages the potential low-rank structures of the unstructured sparse matrices by applying truncated singular value decomposition (SVD), thus converting them into parameter-efficient dense models. Moreover, we introduce nuclear norm regularisation to ensure lower rank and a learnable singular value selection strategy to determine the approximate truncation rate for each matrix. Experiments on multiple speech tasks demonstrate that the proposed method can convert an unstructured sparse model into a light-weight and hardware-friendly dense model with comparable or superior performance.
- Open Access
- 14 July 2023
- Published by Association for Computing Machinery (ACM)
- p. 76-81
Learnable Sparsity Structured Pruning for Acoustic Pre-trained Models
Large-scale pre-trained models bring significant gains to many speech-related tasks. However, it is still challenging to use these large models when computing power of terminal equipment is limited. Pruning is an effective method to reduce memory footprint and cost calculation. The imperfect evaluation criteria of existing pruning methods and the complex fine tuning process result in a relatively high loss of accuracy. To solve these problems, we propose a structured pruning method, which introduced the upper confidence bound of importance scores to assess the potential of each component of the model more accurately. In addition, we also introduce a set of learnable pruning threshold parameters that can be learned via stochastic gradient descent, thereby reducing the hyper-parameter tuning. We apply our method to HuBERT models on automatic speech recognition (ASR) task. Our result shows that for all pruning granularity and pruning ratios, our method yields higher accuracy and speedup ratios in the inference process.When sparsity was 60%, our method performed only 0.63% down.
- 14 July 2023
- Published by Association for Computing Machinery (ACM)
- p. 76-81
Open AccessLearnable Sparsity Structured Pruning for Acoustic Pre-trained Models
Large-scale pre-trained models bring significant gains to many speech-related tasks. However, it is still challenging to use these large models when computing power of terminal equipment is limited. Pruning is an effective method to reduce memory footprint and cost calculation. The imperfect evaluation criteria of existing pruning methods and the complex fine tuning process result in a relatively high loss of accuracy. To solve these problems, we propose a structured pruning method, which introduced the upper confidence bound of importance scores to assess the potential of each component of the model more accurately. In addition, we also introduce a set of learnable pruning threshold parameters that can be learned via stochastic gradient descent, thereby reducing the hyper-parameter tuning. We apply our method to HuBERT models on automatic speech recognition (ASR) task. Our result shows that for all pruning granularity and pruning ratios, our method yields higher accuracy and speedup ratios in the inference process.When sparsity was 60%, our method performed only 0.63% down.
- Open Access
- 14 July 2023
- Published by Association for Computing Machinery (ACM)
- p. 45-50
DistilALHuBERT: A Distilled Parameter Sharing Audio Representation Model
Self-supervised pre-trained audio representation models such as Wav2vec or HuBERT have brought notable improvements to many downstream audio-related tasks, but the huge number of parameters of these pre-trained models sets a barrier to their application on memory-constrained edge devices. Recursive Transformers, represented by Albert, have proven that parameter sharing through transformer layers can obviously reduce the size of pre-trained models while maintaining most of the performance. In this paper, we propose DistilALHuBERT, a lightweight recursive transformer audio representation model distilled from Hubert. Evaluation results on the S3PRL benchmark show that DistilALHuBERT can significantly outperform the DistilHuBERT model with the same number of parameters. Our code and models are available at https://github.com/backspacetg/distilAlhubert.
- 14 July 2023
- Published by Association for Computing Machinery (ACM)
- p. 45-50
Open AccessDistilALHuBERT: A Distilled Parameter Sharing Audio Representation Model
Self-supervised pre-trained audio representation models such as Wav2vec or HuBERT have brought notable improvements to many downstream audio-related tasks, but the huge number of parameters of these pre-trained models sets a barrier to their application on memory-constrained edge devices. Recursive Transformers, represented by Albert, have proven that parameter sharing through transformer layers can obviously reduce the size of pre-trained models while maintaining most of the performance. In this paper, we propose DistilALHuBERT, a lightweight recursive transformer audio representation model distilled from Hubert. Evaluation results on the S3PRL benchmark show that DistilALHuBERT can significantly outperform the DistilHuBERT model with the same number of parameters. Our code and models are available at https://github.com/backspacetg/distilAlhubert.