Memory-Efficient Vision Transformers: An Activation-Aware Mixed-Rank Compression Strategy
Authors:
- Seyedarmin AziziSASeyedarmin Azizi
- Mahdi NazemiMNMahdi Nazemi
- Massoud PedramMPMassoud Pedram
Preprint
Abstract
As Vision Transformers (ViTs) increasingly set new benchmarks in computer vision, their practical deployment on inference engines is often hindered by their significant memory bandwidth and (on-chip) memory footprint requirements. This paper addresses this memory limitation by introducing an activation-aware model compression methodology that uses selective low-rank weight tensor approximations of different layers to reduce the parameter count of ViTs. The key idea is to decompose the weight tensors into a sum of two parameter-efficient tensors while minimizing the error between the product of the input activations with the original weight tensor and the product of the input activations with the approximate tensor sum. This approximation is further refined by adopting an efficient layer-wise error compensation technique that uses the gradient of the layer's output loss. The combination of these techniques achieves excellent results while it avoids being trapped in a shallow local minimum early in the optimization process and strikes a good balance between the model compression and output accuracy. Notably, the presented method significantly reduces the parameter count of DeiT-B by 60% with less than 1% accuracy drop on the ImageNet dataset, overcoming the usual accuracy degradation seen in low-rank approximations. In addition to this, the presented compression technique can compress large DeiT/ViT models to have about the same model size as smaller DeiT/ViT variants while yielding up to 1.8% accuracy gain. These results highlight the efficacy of our approach, presenting a viable solution for embedding ViTs in memory-constrained environments without compromising their performance.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Improved Baselines with Visual Instruction Tuning2024
- SPViT: Enabling Faster Vision Transformers via Latency-Aware Soft Token Pruning2022
- PTQ4ViT: Post-training Quantization for Vision Transformers with Twin Uniform Quantization2022
- TinyViT: Fast Pretraining Distillation for Small Vision Transformers2022
- Segmenter: Transformer for Semantic Segmentation2021
- End-to-End Object Detection with Transformers2020
- HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision2019
- HAQ: Hardware-Aware Automated Quantization With Mixed Precision2019
- On Compressing Deep Models by Low Rank and Sparse Decomposition2017
- The Approximation of One Matrix by Another of Lower RankPsychometrika, 1936