Local Alignment for Medical Vision-Language Pre-Training
- 10 November 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 34 (10577149) , 7321-7334
Abstract
Establishing local semantic correspondences between medical images and their corresponding reports is crucial for effective medical vision-language pre-training. However, existing methods encounter two major challenges: (1) lesion regions in radiological images are often small, blurry, or lack clear boundaries, complicating accurate localization; and (2) medical reports typically contain redundant or non-diagnostic words, hindering precise semantic alignment. To overcome these issues, we propose MedAligner, a specialized local alignment network for medical vision-language pre-training. MedAligner employs dual encoders to extract both global and local representations and uses global contrastive learning to maintain coarse semantic consistency. To enhance local alignment, we introduce a Word-Region Alignment, which generates a learnable word-pixel similarity matrix that is sparsified to identify salient lesion regions accurately. Additionally, our Diagnostic Term Filtering dynamically samples high-importance diagnostic terms from reports, aligning them with identified lesion areas via a local contrastive loss. Importantly, we adopt a progressive training strategy that gradually refines both the input text and semantic alignment. This is achieved by reconstructing concise diagnostic reports and progressively updating word-pixel similarity, generating increasingly accurate image-text pairs. Extensive experiments demonstrate that MedAligner significantly surpasses existing approaches on tasks such as phrase grounding, image-text retrieval, and zero-shot classification, setting new benchmarks in medical vision-language pre-training.Keywords
All Related Versions
Funding Information
- Fundamental Research Program of Shanxi Province (202303021223004)
- National Natural Science Foundation of China (62276159, 62432006)
This publication has 30 references indexed in Scilit:
- CLEFT: Language-Image Contrastive Learning with Efficient Large Language Model and Prompt Fine-Tuning2024
- Enhancing the vision–language foundation model with key semantic knowledge-emphasized report refinementMedical Image Analysis, 2024
- A visual–language foundation model for pathology image analysis using medical TwitterNature Medicine, 2023
- Joint Learning of Localized Representations from Medical Images and Reports2022
- Making the Most of Text Semantics to Improve Biomedical Vision–Language Processing2022
- Generalized radiograph representation learning via cross-supervision between images and free-text radiology reportsNature Machine Intelligence, 2022
- Exploring the effect of image enhancement techniques on COVID-19 detection using chest X-ray imagesComputers in Biology and Medicine, 2021
- MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reportsScientific Data, 2019
- Deep Residual Learning for Image Recognition2016
- Signal, Noise, Signal-to-Noise, and Contrast-to-Noise Ratios2008