Development of Hybrid Deep Learning Systems Combining Convolutional Neural Networks and Vision Transformers for Medical Ultrasound Image Classification
Open Access
- 4 November 2026
- journal article
- research article
- Published by MDPI AG in Diagnostics
- Vol. 16 (21) , 3640
Abstract
Background: Accurate classification of thyroid nodule ultrasound images is clinically significant. This study evaluated whether hybrid architectures combining the local feature extraction of convolutional neural networks (CNNs) with the global context modelling of vision transformers (ViTs) outperform each category in isolation. Methods: Two public datasets (6905 images; 3331 benign, 3574 malignant) were used. Eleven ImageNet-pretrained architectures were compared: CNN-based (ResNet-50, EfficientNet-B4, ConvNeXt), ViT-based (Swin Transformer, DeiT, PVTv2, XCiT) and hybrid (ConViT, CoaT, MobileViT, HuviT). Backbones were frozen and only the classification head was trained; HuviT’s new projection and encoder remained frozen at random initialisation. Wilson and Newcombe 95% confidence intervals (CIs) were computed; explainability used gradient-weighted class activation mapping (Grad-CAM). Results: ConViT achieved the highest accuracy (83.30%; 95% CI 80.91–85.45) and an area under the receiver operating characteristic curve (ROC-AUC) of 89.94% (95% CI 87.8–91.7). ConvNeXt ranked second (82.63%) but had the highest ROC-AUC (90.02%; 95% CI 87.9–91.8); the intervals overlapped. HuviT reached 76.35%. Grad-CAM for a representative case highlighted lesion areas. Conclusions: While ConViT achieved the highest performance on threshold-based metrics, these findings suggest the potential of specific hybrid designs rather than establishing general superiority of hybrid architectures. Given the retrospective design and the partial association between image source and class label, prospective external validation is warranted.Keywords
Funding Information
- Inönü Üniversitesi (TSA-2026-5058)
This publication has 40 references indexed in Scilit:
- LGF‐Net: A multi‐scale feature fusion network for thyroid nodule ultrasound image classificationJournal of Applied Clinical Medical Physics, 2025
- Exploring the synergies of hybrid convolutional neural network and Vision Transformer architectures for computer vision: A surveyEngineering Applications of Artificial Intelligence, 2025
- ViTs as backbones: Leveraging vision transformers for feature extractionInformation Fusion, 2025
- Multi-class classification of thyroid nodules from automatic segmented ultrasound images: Hybrid ResNet based UNet convolutional neural network approachComputer Methods and Programs in Biomedicine, 2023
- Automatic classification of ultrasound thyroids images using vision transformers and generative adversarial networksScientific African, 2023
- PVT v2: Improved baselines with pyramid vision transformerComputational Visual Media, 2022
- Comparative analysis of image classification algorithms based on traditional machine learning and deep learningPattern Recognition Letters, 2021
- A survey of the recent architectures of deep convolutional neural networksArtificial Intelligence Review, 2020
- ACR Thyroid Imaging, Reporting and Data System (TI-RADS): White Paper of the ACR TI-RADS CommitteeJournal of the American College of Radiology, 2017
- Confidence intervals for an effect size measure based on the Mann–Whitney statistic. Part 2: asymptotic methods and evaluationStatistics in Medicine, 2005