Adaptation-Preserving Structured Pruning for Efficient Transfer Learning in Deep Neural Networks
Open Access
Abstract
Transfer learning enables pretrained deep networks to adapt to new tasks but typically preserves their original computational structure, leaving much of the inference cost unchanged. Structured pruning can reduce this complexity; however, introducing sparsity during fine-tuning creates a competing optimization problem in which transferred representations must adapt while the network simultaneously determines which structures can be removed. We propose GASP, an adaptation-preserving structured pruning framework that treats task adaptation and structural sparsity learning as a coordinated optimization process, allowing for the sparse structure to emerge progressively as pretrained representations adapt to the target task. Rather than imposing structural decisions independently of this adaptation, GASP first establishes target-task adaptation through a warm-up phase, before progressively introducing sparsity pressure through an equation-driven schedule. In parallel, temperature annealing gradually sharpens input-aware continuous gates toward increasingly differentiated structural responses. Together, these mechanisms regulate the onset and progression of sparsification while structural importance evolves during fine-tuning. The learned gating responses are subsequently aggregated over the validation data and converted into a physically compact architecture through global Otsu-based thresholding and structural reconstruction, avoiding predefined sparsity ratios and iterative prune–retrain cycles. The same optimization principle is applied to channels in CNNs and MLP units in Swin Transformers without additional trainable mask parameters. Experiments with VGG16, EfficientNet-B0, and Swin-Tiny on CIFAR-10, PlantVillage, and PlantDoc show that GASP achieves accuracy improvements of up to 2.22%, with the corresponding configuration reducing parameters and FLOPs by 91.37% and 88.83%, respectively. Raspberry Pi 4 deployment further demonstrates that the learned structural sparsity translates into practical inference efficiency. Overall, the results demonstrate the effectiveness of coordinating representation adaptation and structural sparsification when deriving compact task-adapted models.Keywords
This publication has 28 references indexed in Scilit:
- Deformable transformer framework for automated calf respiratory health scoringComputers and Electronics in Agriculture, 2026
- An end-to-end vision–language framework with LLM-based reasoning for decision-oriented tomato leaf disease management in greenhouse environmentsComputers and Electronics in Agriculture, 2026
- RyNet: Multi-Level Attention for Rich and Complementary Feature Representation in CNNsIEEE Access, 2025
- Robust Real-Time Arabic Speech Recognition for AAVs in Adverse Acoustic Conditions Using Lightweight CNNsIEEE Access, 2025
- Dynamic attention-based transfer learning strategies for efficient and scalable visual recognition in resource-constrained environmentsApplied Soft Computing, 2025
- Channel Pruning of Transfer Learning Models Using Novel TechniquesIEEE Access, 2024
- Convolutional neural network pruning based on multi-objective feature map selection for image classificationApplied Soft Computing, 2023
- RFPruning: A retraining-free pruning method for accelerating convolutional neural networksApplied Soft Computing, 2021
- AutoPruner: An end-to-end trainable filter pruning method for efficient deep model inferencePattern Recognition, 2020
- An overview of statistical learning theoryIEEE Transactions on Neural Networks, 1999