Frustratingly Easy Task-aware Pruning for Large Language Models
Preprint
Abstract
Pruning provides a practical solution to reduce the resources required to run large language models (LLMs) to benefit from their effective capabilities as well as control their cost for training and inference. Research on LLM pruning often ranks the importance of LLM parameters using their magnitudes and calibration-data activations and removes (or masks) the less important ones, accordingly reducing LLMs' size. However, these approaches primarily focus on preserving the LLM's ability to generate fluent sentences, while neglecting performance on specific domains and tasks. In this paper, we propose a simple yet effective pruning approach for LLMs that preserves task-specific capabilities while shrinking their parameter space. We first analyze how conventional pruning minimizes loss perturbation under general-domain calibration and extend this formulation by incorporating task-specific feature distributions into the importance computation of existing pruning algorithms. Thus, our framework computes separate importance scores using both general and task-specific calibration data, partitions parameters into shared and exclusive groups based on activation-norm differences, and then fuses their scores to guide the pruning process. This design enables our method to integrate seamlessly with various foundation pruning techniques and preserve the LLM's specialized abilities under compression. Experiments on widely used benchmarks demonstrate that our approach is effective and consistently outperforms the baselines with identical pruning ratios and different settings.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Wanda++: Pruning Large Language Models via Regional Gradients2025
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints2023
- Dynamic Sparse Training for Deep Reinforcement Learning2022
- Training Language Models to Follow Instructions with Human Feedback2022
- Training Compute-Optimal Large Language Models2022
- What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical ExamsApplied Sciences, 2021
- The promise of machine learning in predicting treatment outcomes in psychiatryWorld Psychiatry, 2021
- Structured Pruning of Large Language Models2020
- Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge ComputingProceedings of the IEEE, 2019
- Perplexity—a measure of the difficulty of speech recognition tasksThe Journal of the Acoustical Society of America, 1977