Numerical Pruning for Efficient Autoregressive Models
Authors:
Preprint
Abstract
Transformers have emerged as the leading architecture in deep learning, proving to be versatile and highly effective across diverse domains beyond language and image processing. However, their impressive performance often incurs high computational costs due to their substantial model size. This paper focuses on compressing decoder-only transformer-based autoregressive models through structural weight pruning to improve the model efficiency while preserving performance for both language and image generation tasks. Specifically, we propose a training-free pruning method that calculates a numerical score with Newton's method for the Attention and MLP modules, respectively. Besides, we further propose another compensation algorithm to recover the pruned model for better performance. To verify the effectiveness of our method, we provide both theoretical support and extensive experiments. Our experiments show that our method achieves state-of-the-art performance with reduced memory usage and faster generation speeds on GPUs.Keywords
All Related Versions
This publication has 36 references indexed in Scilit:
- HotaQ: Hardware Oriented Token Adaptive Quantization for Large Language ModelsIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2024
- Condense: A Framework for Device and Frequency Adaptive Neural Network Models on the Edge2023
- Compiler-Aware Neural Architecture Search for On-Mobile Real-time Super-Resolution2022
- SPViT: Enabling Faster Vision Transformers via Latency-Aware Soft Token Pruning2022
- Solving SDP Faster: A Robust IPM Framework and Efficient Implementation2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Radio Frequency Fingerprinting on the EdgeIEEE Transactions on Mobile Computing, 2021
- Unifying Matrix Data Structures: Simplifying and Speeding up Iterative Algorithms2021
- Toward Moderate Overparameterization: Global Convergence Guarantees for Training Shallow Neural NetworksIEEE Journal on Selected Areas in Information Theory, 2020
- Acceleration of Stochastic Approximation by AveragingSIAM Journal on Control and Optimization, 1992