MPACT: Multimodal Pulmonary Abnormality Classification with CNN-Transformer and Content-Aware Sampling

Authors:

Abstract
Pulmonary diseases continue to be among the largest causes of morbidity and mortality globally and are a significant strain on health care systems, particularly in low- and middle-income countries. Chest X-ray is a primary imaging modality, but image interpretation is not solely dependent on the image features, but on the report text combined with image interpretation. Current deep learning systems are mainly unimodal, and even the recent multimodal ones usually concatenate image and text features at shallow levels, failing to meaningfully influence their representation. In this work, we introduce MPACT, a multimodal method for classifying binary pulmonary abnormalities in pairs of radiographs and radiology reports. The proposed framework includes: (i) a parallel hybrid visual encoder designed to obtain the local texture and global structural patterns simultaneously, consisting of a custom CNN backbone and Swin Transformer branch; (ii) a domain-adapted BioBERT text encoder with handcrafted auxiliary clinical features extracted from the text of radiology reports; (iii) a bidirectional cross-modal attention fusion mechanism; (iv) a classification head that is activated by GELU and has a residual skip-connection branch; and (v) a content-aware weighted sampling strategy, which considers class imbalance and within-class clinical heterogeneity. Compared with four benchmark models, MPACT outperforms them in terms of accuracy (98.69%), precision (0.9888), specificity (0.9887), sensitivity (0.9851), F1-score (0.9870), and AUC-ROC (0.9987) on the Indiana University Chest X-Ray (IU-XRay) dataset for distinguishing between normal and abnormal cases. Each component of the architecture contributes uniquely, as confirmed by a systematic ablation study.