Rethinking Vision Transformer Depth via Structural Reparameterization
Preprint
Abstract
The computational overhead of Vision Transformers in practice stems fundamentally from their deep architectures, yet existing acceleration strategies have primarily targeted algorithmic-level optimizations such as token pruning and attention speedup. This leaves an underexplored research question: can we reduce the number of stacked transformer layers while maintaining comparable representational capacity? To answer this, we propose a branch-based structural reparameterization technique that operates during the training phase. Our approach leverages parallel branches within transformer blocks that can be systematically consolidated into streamlined single-path models suitable for inference deployment. The consolidation mechanism works by gradually merging branches at the entry points of nonlinear components, enabling both feed-forward networks (FFN) and multi-head self-attention (MHSA) modules to undergo exact mathematical reparameterization without inducing approximation errors at test time. When applied to ViT-Tiny, the framework successfully reduces the original 12-layer architecture to 6, 4, or as few as 3 layers while maintaining classification accuracy on ImageNet-1K. The resulting compressed models achieve inference speedups of up to 37% on mobile CPU platforms. Our findings suggest that the conventional wisdom favoring extremely deep transformer stacks may be unnecessarily restrictive, and point toward new opportunities for constructing efficient vision transformers.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition2024
- SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications2023
- NetBooster: Empowering Tiny Deep Learning By Standing on the Shoulders of Deep Giants2023
- X-Pruner: eXplainable Pruning for Vision Transformers2023
- Self-slimmed Vision Transformer2022
- EdgeViTs: Competing Light-Weight CNNs on Mobile Devices with Vision Transformers2022
- PVT v2: Improved baselines with pyramid vision transformerComputational Visual Media, 2022
- Diverse Branch Block: Building a Convolution as an Inception-like Unit2021
- MultiNet++: Multi-Stream Feature Aggregation and Geometric Loss Strategy for Multi-Task Learning2019
- Deep learning with coherent nanophotonic circuitsNature Photonics, 2017