Cross-Modal Causal Representation Learning for Radiology Report Generation
Top Cited Papers
- 16 May 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 34 (10577149) , 2970-2985
Abstract
Radiology Report Generation (RRG) is essential for computer-aided diagnosis and medication guidance, which can relieve the heavy burden of radiologists by automatically generating the corresponding radiology reports according to the given radiology image. However, generating accurate lesion descriptions remains challenging due to spurious correlations from visual-linguistic biases and inherent limitations of radiological imaging, such as low resolution and noise interference. To address these issues, we propose a two-stage framework named Cross-Modal Causal Representation Learning (CMCRL), consisting of the Radiological Cross-modal Alignment and Reconstruction Enhanced (RadCARE) pre-training and the Visual-Linguistic Causal Intervention (VLCI) fine-tuning. In the pre-training stage, RadCARE introduces a degradation-aware masked image restoration strategy tailored for radiological images, which reconstructs high-resolution patches from low-resolution inputs to mitigate noise and detail loss. Combined with a multiway architecture and four adaptive training strategies (e.g., text postfix generation with degraded images and text prefixes), RadCARE establishes robust cross-modal correlations even with incomplete data. In the VLCI phase, we deploy causal front-door intervention through two modules: the Visual Deconfounding Module (VDM) disentangles local-global features without fine-grained annotations, while the Linguistic Deconfounding Module (LDM) eliminates context bias without external terminology databases. Experiments on IU-Xray and MIMIC-CXR show that our CMCRL pipeline significantly outperforms state-of-the-art methods, with ablation studies confirming the necessity of both stages. Code and models are available at.Keywords
All Related Versions
Funding Information
- National Key Research and Development Program of China (2021ZD0111601)
- National Natural Science Foundation of China (62436009, 62322608, 62301532)
- Basic and Applied Basic Research Foundation of Guangdong Province (2025A1515011874, 2023A1515011530)
This publication has 67 references indexed in Scilit:
- ShareGPT4V: Improving Large Multi-modal Models with Better Captions2024
- R2GenGPT: Radiology Report Generation with frozen LLMsMeta-Radiology, 2023
- Cross-Modal Prototype Driven Network for Radiology Report Generation2022
- Causal Reasoning Meets Visual Representation Learning: A Prospective StudyMachine Intelligence Research, 2022
- GRIT: Faster and Better Image Captioning Transformer Using Dual Visual Features2022
- Lesion Guided Explainable Few Weak-Shot Medical Report Generation2022
- Knowledge matters: Chest radiology report generation with general and specific knowledgeMedical Image Analysis, 2022
- AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation2021
- Preparing a collection of radiology examinations for distribution and retrievalJournal of the American Medical Informatics Association, 2015
- Deep visual-semantic alignments for generating image descriptions2015