Shenghuan He
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Shenghuan He has a total of 6 co-authors. Here are the top co-authors by number of shared publications:
- Hao Yi
Alibaba Group, Hangzhou, Zhejiang, China
1 shared publication - Hong Liu
Alibaba Group, Hangzhou, Zhejiang, China
1 shared publication - Qinghui Sun
Alibaba Group, Hangzhou, Zhejiang, China
1 shared publication - Weilun Wang
Alibaba Group, Hangzhou, Zhejiang, China
1 shared publication - Yanni Zhu
Alibaba Group, Hangzhou, Zhejiang, China
1 shared publication - Zehua Cai
Alibaba Group, Hangzhou, Zhejiang, China
1 shared publication
Latest Publications
- Open Access
- 24 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 2775-2783
Dual-Assessment Driven Pruning: Iterative Optimizing Layer-wise Sparsity for Large Language Model
Large Language Models (LLMs) have demonstrated efficacy in various domains, but deploying these models is economically challenging due to extensive parameter counts. Numerous efforts have been dedicated to reducing the parameter count of these models without compromising performance, employing a technique known as model pruning. Conventional pruning methods assess the significance of weights within individual layers and typically apply uniform sparsity levels across all layers, potentially neglecting the varying significance of each layer. To address this oversight, we first propose a dual-assessment driven pruning strategy that employs both intra-layer metric and global performance metric to comprehensively evaluate the impact of pruning. Then our method leverages an iterative optimization algorithm to find the optimal layer-wise sparsity distribution, thereby minimally impacting model performance. Extensive benchmark evaluations on state-of-the-art LLM architectures such as LLaMAv2 and OPT across a variety of NLP tasks demonstrate the effectiveness of our approach. When applied to the LLaMaV2-7B model with an overall pruning sparsity of 80%, our method achieves a 50% reduction in perplexity compared to the benchmark. The results indicate that our method significantly outperforms existing state-of-the-art methods in preserving performance after pruning.
- 24 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 2775-2783
Open AccessDual-Assessment Driven Pruning: Iterative Optimizing Layer-wise Sparsity for Large Language Model
Large Language Models (LLMs) have demonstrated efficacy in various domains, but deploying these models is economically challenging due to extensive parameter counts. Numerous efforts have been dedicated to reducing the parameter count of these models without compromising performance, employing a technique known as model pruning. Conventional pruning methods assess the significance of weights within individual layers and typically apply uniform sparsity levels across all layers, potentially neglecting the varying significance of each layer. To address this oversight, we first propose a dual-assessment driven pruning strategy that employs both intra-layer metric and global performance metric to comprehensively evaluate the impact of pruning. Then our method leverages an iterative optimization algorithm to find the optimal layer-wise sparsity distribution, thereby minimally impacting model performance. Extensive benchmark evaluations on state-of-the-art LLM architectures such as LLaMAv2 and OPT across a variety of NLP tasks demonstrate the effectiveness of our approach. When applied to the LLaMaV2-7B model with an overall pruning sparsity of 80%, our method achieves a 50% reduction in perplexity compared to the benchmark. The results indicate that our method significantly outperforms existing state-of-the-art methods in preserving performance after pruning.