MoR–Swin: Efficient Vision Transformer Using Mixture of Recursions
Open Access
- 12 August 2026
- journal article
- research article
- Published by MDPI AG in Journal of Imaging
- Vol. 12 (8) , 377
Abstract
Vision Transformers, especially Swin Transformer, have become default backbones for various vision tasks but suffer from high memory consumption and training costs. This letter proposes MoR–Swin, a novel architecture that integrates Mixture of Recursions (MoR) into Swin Transformer. An adaptive token-level recursion mechanism dynamically allocates computational depth based on semantic complexity. A recursive window attention module and a lightweight router with load balancing loss are introduced. Extensive experiments on ImageNet classification, COCO detection, and ADE20K segmentation show that MoR–Swin reduces parameters by about 50% and accelerates inference up to twofold at a modest accuracy cost (within about 0.5 points of Swin-B on ImageNet-1K). It provides a new technical pathway for optimizing Vision Transformer models, significantly enhancing their applicability in resource-constrained environments.Keywords
Funding Information
- National Natural Science Foundation of China (12502326)
This publication has 13 references indexed in Scilit:
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your PhonearXiv, 2024
- Mixtral of ExpertsarXiv, 2024
- GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsarXiv, 2022
- DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI ScalearXiv, 2022
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient SparsityarXiv, 2022
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScalearXiv, 2021
- GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingarXiv, 2020
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerarXiv, 2017
- Adaptive Mixtures of Local ExpertsNeural Computation, 1991
- Backpropagation Applied to Handwritten Zip Code RecognitionNeural Computation, 1989