Bridging Medical Imaging and Reports:Learning Radiologist's Nuances via Fine-Grained Multi-Modal Alignment
Preprint
- 25 March 2025
- preprint
- Published by Springer Nature in Research Square
Abstract
Precise and explainable alignment between data from different data modalities is crucial for advancing artificial general intelligence in medicine. In this work, we present CAMMAL (Cyclic Adaptive Medical Modality ALignment), a framework that can achieve fine-grained vision-language alignment through two key innovations, including an Adaptive Patch-Word Matching (AdaMatch) mechanism that dynamically correlates regions in medical images with specific words in radiology reports, and a bidirectional generative architecture that leverages the alignment between textual and visual codebooks to guide the translation between modalities within a single model. Evaluation of CAMMAL on the chest X-rays (combined MIMIC-CXR and OpenI datasets) and mammography (EMBED dataset) demonstrates its superior performance across multiple metrics over other methods. Human reader studies by radiologists validate the clinical effectiveness of the generated reports and synthetic images, particularly in capturing anatomical structures (73% rated good/excellent) and meaningful findings (56% rated very good/excellent). By enabling systematic capture of inter-modality medical data correspondence and fluid multi-way translation between modalities, CAMMAL could advance the development of interpretable and clinically reliable AI systems by bridging the gap between medical images and textual descriptions. Its ability to accurately retrieve, generate, and align reports with imaging data offers potential improvements in learning detailed radiologists' knowledge, which could be vital for AI-assisted diagnosis and medical education. The bidirectional generative approach further enhances model transparency and explainability, fostering trust for AI-driven healthcare applications. As multimodal medical foundation models continue to evolve, CAMMAL provides a foundation for integrating vision and language understanding, with the potential for broader clinical applications beyond radiology.Keywords
All Related Versions
This publication has 50 references indexed in Scilit:
- Matching patients to clinical trials with large language modelsNature Communications, 2024
- Enhancing representation in radiography-reports foundation model: a granular alignment algorithm using masked contrastive learningNature Communications, 2024
- Radiology report generation with a learned knowledge base and multi-modal alignmentMedical Image Analysis, 2023
- Cross-Modal Prototype Driven Network for Radiology Report Generation2022
- Structured reporting in radiology: a systematic review to explore its potentialEuropean Radiology, 2021
- Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment2021
- Bridging the gap: interactive, case-based learning in radiology educationAbdominal Radiology, 2021
- Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessment2020
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks2020
- UNITER: UNiversal Image-TExt Representation Learning2020