Abstract
Image fusion involves integrating the information of multiple source images into one fused image, maximizing each source image's advantages. Existing image fusion methods combining text information mainly focus on enhancing the fusion effect, ignoring the requirements of subsequent tasks. High level task oriented fusion methods are usually limited to image information processing, ignoring the potential of text information at the high-level semantic, such as scene understanding and target recognition. We propose an image fusion method called TextFuse, which combines text information and subsequent task requirements, to guide image fusion through the text description generated by the source image and subsequent task requirements. Specifically, we use the text generation model to generate text descriptions for the source image, and design strategies to generate additional text information for key areas to comprehensively express the content of the source image. In the fusion process, we use text information and detection network to guide the model, so as to enhance the model's understanding of the image, generate a fusion image containing rich semantic information and visual details, and improve the performance in subsequent tasks. The text information provides additional semantic context for the image fusion process, enhances the details of key areas, and significantly improves the quality of the fused image. At the same time, the guidance of subsequent tasks further optimized the performance of fused images in follow-up tasks. Experimental results show that the proposed method is significantly better than the state-of-the-art image fusion methods in terms of visual effect and follow-up task performance. The source code is available at: https://github.com/poison-pig/TextFuse.
Funding Information
  • Shandong Provincial Natural Science Foundation (ZR2024QF115)
  • National Natural Science Foundation of China (62406155)
  • Innovation Capability Enhancement Project for Technology based Small and Medium-sized Enterprises of Shandong Province (2025TSGCCZZB0077)

This publication has 50 references indexed in Scilit: