A Hybrid Learnable Fusion of ConvNeXt and Swin Transformer for Optimized Image Classification
Open Access
Abstract
Medical image classification often relies on CNNs to capture local details (e.g., lesions, nodules) or on transformers to model long-range dependencies. However, each paradigm alone is limited in addressing both fine-grained structures and broader anatomical context. We propose ConvTransGFusion, a hybrid model that fuses ConvNeXt (for refined convolutional features) and Swin Transformer (for hierarchical global attention) using a learnable dual-attention gating mechanism. By aligning spatial dimensions, scaling each branch adaptively, and applying both channel and spatial attention, the proposed architecture bridges local and global representations, melding fine-grained lesion details with the broader anatomical context essential for accurate diagnosis. Tested on four diverse medical imaging datasets—including X-ray, ultrasound, and MRI scans—the proposed model consistently achieves superior accuracy, precision, recall, F1, and AUC over state-of-the-art CNNs and transformers. Our findings highlight the benefits of combining convolutional inductive biases and transformer-based global context in a single learnable framework, positioning ConvTransGFusion as a robust and versatile solution for real-world clinical applications.Keywords
This publication has 44 references indexed in Scilit:
- Multi-scale feature fusion and class weight loss for skin lesion classificationComputers in Biology and Medicine, 2024
- ConvMedSegNet: A multi-receptive field depthwise convolutional neural network for medical image segmentationComputers in Biology and Medicine, 2024
- CECT: Controllable ensemble CNN and transformer for COVID-19 image classificationComputers in Biology and Medicine, 2024
- Diagnosis of Alzheimer's disease via optimized lightweight convolution-attention and structural MRIComputers in Biology and Medicine, 2024
- Do it the transformer way: A comprehensive review of brain and vision transformers for autism spectrum disorder diagnosis and classificationComputers in Biology and Medicine, 2023
- HAU-Net: Hybrid CNN-transformer for breast ultrasound image segmentationBiomedical Signal Processing and Control, 2023
- Hybrid CNN-Transformer model for medical image segmentation with pyramid convolution and multi-layer perceptronBiomedical Signal Processing and Control, 2023
- Recent progress in transformer-based medical image analysisComputers in Biology and Medicine, 2023
- MedViT: A robust vision transformer for generalized medical image classificationComputers in Biology and Medicine, 2023
- Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep LearningCell, 2018