Infrared and Visible Image Fusion With Language-Driven Loss and Knowledge Distillation
- 1 January 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 37 (10) , 1-15
Abstract
Infrared and visible image fusion (IVIF) has attracted much attention owing to the highly complementary properties of the two image modalities. Due to the lack of ground-truth fused images, the fusion output of current deep-learning-based methods heavily depends on the loss functions defined mathematically. As it is hard to well mathematically define the fused image without ground truth, the performance of existing fusion methods is limited. In this article, we first propose to use natural language to express the objective of IVIF, which can avoid the explicit mathematical modeling of fusion output in current losses, and make full use of the advantage of language expression to improve the fusion performance. For this purpose, we present a comprehensive language-expressed fusion objective and encode relevant texts into the multimodal embedding space using CLIP. A language-driven fusion model is then constructed in the embedding space by establishing the relationship among the embedded vectors representing the fusion objective and input image modalities. Finally, a language-driven loss is derived to make the actual IVIF aligned with the embedded language-driven fusion model via supervised training. In addition, to ensure that the loss works well with high robustness and generalization in practice, we introduce a novel regularization and patch filtering approach in the training. For real-time applications, we further propose a novel knowledge distillation framework, in which the student conducts self-learning under the supervision of the teacher that has been trained with the language-driven loss. This enables the student to maintain comparable performance and, more importantly, achieve real-time inference even on high-resolution images. Experiments show that our method can obtain much better fusion results than existing techniques. The code is available at https://github.com/wyhlaowang/LDFusion.Keywords
Funding Information
- National Natural Science Foundation of China (62471045)
- National Natural Science Foundation of China (62071036)
This publication has 69 references indexed in Scilit:
- A Deep Learning Framework for Infrared and Visible Image Fusion Without Strict RegistrationInternational Journal of Computer Vision, 2023
- ReCoNet: Recurrent Correction Network for Fast and Efficient Multi-modality Image Fusion2022
- CUFD: An encoder–decoder network for visible and infrared image fusion based on common and unique feature decompositionComputer Vision and Image Understanding, 2022
- RXDNFuse: A aggregated residual dense network for infrared and visible image fusionInformation Fusion, 2021
- FusionGAN: A generative adversarial network for infrared and visible image fusionInformation Fusion, 2019
- Multimodal Unsupervised Image-to-Image Translation2018
- The TNO Multiband Image Data CollectionData in Brief, 2017
- Perceptual fusion of infrared and visible images through a hybrid multi-scale decomposition with Gaussian and bilateral filtersInformation Fusion, 2016
- Microsoft COCO: Common Objects in Context2014
- A new image fusion performance metric based on visual information fidelityInformation Fusion, 2013