3 × 3 Kernel is ALL You Need for Vision
- 21 April 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. PP (10518215) , 1
Abstract
Most modern Convolutional Neural Networks (CNNs) employ a multi-branch structure with various-sized convolutions to capture long- and short-range dependencies. However, these CNNs use large kernel convolutions (e.g., astonishingly 101 kernels) and specialized techniques (e.g., reparameterization and sparsity), increasing complexity in both training and inference stages. This paper focuses on designing an efficient CNN based on pure 3×3 convolutions without introducing complex operations and techniques. Specifically, we propose a Spatial Pyramid (SP) block, which consists of the Multi-branch Residual (MbR) module and the Gated-branch Residual (GbR) module. The MbR introduces multiscale pooling as the key component, thus capturing long-range visual cues through large down-sampling rates and shorter-range dependencies through low down-sampling rates while maintaining low computational complexity. Besides, the GbR uses one 3×3 convolution to refine dependencies along spatial and channel dimensions. Based on the SP block, we construct the Spatial Pyramid CNN (SPCNN), a model composed exclusively of Point-Wise Convolution and 3×3 Depth-Wise Convolution. Under comparable computational complexity, SPCNN significantly outperforms the state-of-the-art CNN PeLK (83.6% vs 82.6%) with only 3 × 3 kernels (compared to 101 × 101 kernels in PeLK). Besides, our SPCNN demonstrates comparability with state-of-the-art backbones in lightweight models, object detection, instance segmentation, and semantic segmentation. Moreover, evaluations of four image retrieval benchmarks also demonstrate the effectiveness. All codes are released at https://github.com/xiaolai-sqlai/SPCNN.Keywords
Funding Information
- National Natural Science Foundation of China (62273301)
This publication has 67 references indexed in Scilit:
- DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs2024
- FMGNet: An efficient feature-multiplex group network for real-time vision taskPattern Recognition, 2024
- Visual attention networkComputational Visual Media, 2023
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design2018
- Unified Perceptual Parsing for Scene Understanding2018
- A Deep Learning-Based Approach to Progressive Vehicle Re-identification for Urban Surveillance2016
- Deep Networks with Stochastic Depth2016
- Microsoft COCO: Common Objects in Context2014
- Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition2014
- Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories2006