Harnessing Text Insights With Visual Alignment for Medical Image Segmentation
- 21 August 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 45 (2) , 477-489
Abstract
Pre-trained vision-language models (VLMs) and language models (LMs) have recently garnered significant attention due to their remarkable ability to represent textual concepts, opening up new avenues in vision tasks. In medical image segmentation, efforts are being made to integrate text and image data using VLMs and LMs. However, current text-enhanced approaches face several challenges. First, using separate pre-trained vision and text models to encode image and text data can result in semantic shifts. Second, while VLMs can establish the correspondence between visual and textual features when pre-trained on paired image-text data, this alignment often deteriorates during segmentation tasks due to misalignment between the text and vision components in ongoing learning. In this paper, we propose TeViA, a novel approach that seamlessly integrates with various vision and text models, irrespective of their pre-training relationships. This integration is achieved through a segmentation-specific text-to-vision alignment design, ensuring both information gain and semantic consistency. Specifically, for each training data, a foreground visual representation is extracted from the segmentation head and used to supervise projection layers, thereby adjusting the textual features to better contribute to the segmentation task. Additionally, a historic visual prototype is created by aggregating target semantics from all training data and is updated using a momentum-based manner. This prototype aims to enhance the visual representation of each data instance by establishing feature-level connections, which in turn refines the textual features. The superiority of TeViA is validated on five public datasets, exhibiting over 6% Dice improvements compared to vision-only methods. Code is available at: https://github.com/jgfiuuuu/TeViA.Keywords
Funding Information
- National Natural Science Foundation of China (62171377)
- National Natural Science Foundation of China (92470101)
- “Pioneer” and “Leading Goose” R&D Program of Zhejiang, China (2025C01201 (SD2))
- Innovation Foundation for Doctor Dissertation of Northwestern Polytechnical University (CX2025019)
This publication has 71 references indexed in Scilit:
- ACC-UNet: A Completely Convolutional UNet Model for the 2020s2023
- Text-Guided Cross-Position Attention for Segmentation: Case of Medical Image2023
- CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training2023
- Ariadne’s Thread: Using Text Prompts to Improve Segmentation of Infected Areas from Chest X-ray Images2023
- TPRO: Text-Prompting-Based Weakly Supervised Histopathology Tissue Segmentation2023
- Swin-Unet: Unet-Like Pure Transformer for Medical Image Segmentation2023
- Making the Most of Text Semantics to Improve Biomedical Vision–Language Processing2022
- UniMiSS: Universal Medical Self-supervised Learning via Breaking Dimensionality Barrier2022
- Transformer Based Multiple Instance Learning for Weakly Supervised Histopathology Image Segmentation2022
- UNet++: A Nested U-Net Architecture for Medical Image Segmentation2018