A Radiology Report Generation Model Based on Boosting Cross-Modal Information Joint Memory Alignment
- 27 February 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 33 (10709908) , 1091-1095
Abstract
Radiology Report Generation (RRG) is designed to improve the quality of radiology reports, reduce the workload and subjective errors of radiologists, and facilitate clinical automation. Previous approaches have typically used encoder-decoder architectures, mostly by injecting additional auxiliary information (e.g., knowledge graphs etc.) into the model to improve model performance; few studies have explored cross-modal information interaction and memory utilization. Here, we present a boosted cross-modal information joint memory alignment (BcI-mA) Transformer to efficiently generate radiology report tasks. First, we design a cross-modal information boost module (BcI), which can effectively enhance the interaction between intra- and inter-frame vector features and can be dynamically and iteratively updated with the training process as auxiliary information. Second, a plug-and-play module, which consists of BcI and the memory-aligned (mA) attention mechanism, is designed and embedded into BcI-mA. The method flexibly injects auxiliary information extracted from BcI into mA and utilizes it for information alignment with historical memory. The experimental results show that BcI-mA achieves substantial improvements and outperforms the latest state-of-the-art methods in benchmarking two popular radiology reporting datasets. Further analysis demonstrated that our method is capable of generating radiology reports that contain the necessary medical anomaly information terms.Keywords
Funding Information
- National Natural Science Foundation of China (61966026)
- Natural Science Foundation of Inner Mongolia Autonomous Region (2020MS06015)
This publication has 20 references indexed in Scilit:
- Dual-modality visual feature flow for medical report generationMedical Image Analysis, 2024
- A Novel Deep Learning Model for Medical Report Generation by Inter-Intra Information CalibrationIEEE Journal of Biomedical and Health Informatics, 2023
- Cross-Modal Prototype Driven Network for Radiology Report Generation2022
- Knowledge matters: Chest radiology report generation with general and specific knowledgeMedical Image Analysis, 2022
- Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation2021
- MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reportsScientific Data, 2019
- Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation2016
- Deep Residual Learning for Image Recognition2016
- Preparing a collection of radiology examinations for distribution and retrievalJournal of the American Medical Informatics Association, 2015
- CIDEr: Consensus-based image description evaluation2015