TextFuse: Text Information Guided Infrared and Visible Image Fusion
- 12 March 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. PP (15209210) , 1-15
Abstract
Image fusion involves integrating the information of multiple source images into one fused image, maximizing each source image's advantages. Existing image fusion methods combining text information mainly focus on enhancing the fusion effect, ignoring the requirements of subsequent tasks. High level task oriented fusion methods are usually limited to image information processing, ignoring the potential of text information at the high-level semantic, such as scene understanding and target recognition. We propose an image fusion method called TextFuse, which combines text information and subsequent task requirements, to guide image fusion through the text description generated by the source image and subsequent task requirements. Specifically, we use the text generation model to generate text descriptions for the source image, and design strategies to generate additional text information for key areas to comprehensively express the content of the source image. In the fusion process, we use text information and detection network to guide the model, so as to enhance the model's understanding of the image, generate a fusion image containing rich semantic information and visual details, and improve the performance in subsequent tasks. The text information provides additional semantic context for the image fusion process, enhances the details of key areas, and significantly improves the quality of the fused image. At the same time, the guidance of subsequent tasks further optimized the performance of fused images in follow-up tasks. Experimental results show that the proposed method is significantly better than the state-of-the-art image fusion methods in terms of visual effect and follow-up task performance. The source code is available at: https://github.com/poison-pig/TextFuse.Keywords
Funding Information
- Shandong Provincial Natural Science Foundation (ZR2024QF115)
- National Natural Science Foundation of China (62406155)
- Innovation Capability Enhancement Project for Technology based Small and Medium-sized Enterprises of Shandong Province (2025TSGCCZZB0077)
This publication has 50 references indexed in Scilit:
- Depicting Beyond Scores: Advancing Image Quality Assessment Through Multi-modal Language Models2024
- Rethinking the necessity of image fusion in high-level vision tasks: A practical infrared and visible image fusion network based on progressive semantic injection and scene fidelityInformation Fusion, 2023
- DIVFusion: Darkness-free infrared and visible image fusionInformation Fusion, 2022
- PIAFusion: A progressive infrared and visible image fusion network based on illumination awareInformation Fusion, 2022
- Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion networkInformation Fusion, 2022
- FusionGAN: A generative adversarial network for infrared and visible image fusionInformation Fusion, 2019
- A new image quality metric for image fusion: The sum of the correlations of differencesAEU - International Journal of Electronics and Communications, 2015
- A Review of Quality Metrics for Fused ImageAquatic Procedia, 2015
- A new image fusion performance metric based on visual information fidelityInformation Fusion, 2013
- A non-reference image fusion metric based on mutual information of image featuresComputers and Electrical Engineering, 2011