MaeFuse: Transferring Omni Features With Pretrained Masked Autoencoders for Infrared and Visible Image Fusion via Guided Training
Top Cited Papers
- 19 February 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 34 (10577149) , 1340-1353
Abstract
In this paper, we introduce MaeFuse, a novel autoencoder model designed for Infrared and Visible Image Fusion (IVIF). The existing approaches for image fusion often rely on training combined with downstream tasks to obtain high-level visual information, which is effective in emphasizing target objects and delivering impressive results in visual quality and task-specific applications. Instead of being driven by downstream tasks, our model called MaeFuse utilizes a pretrained encoder from Masked Autoencoders (MAE), which facilities the omni features extraction for low-level reconstruction and high-level vision tasks, to obtain perception friendly features with a low cost. In order to eliminate the domain gap of different modal features and the block effect caused by the MAE encoder, we further develop a guided training strategy. This strategy is meticulously crafted to ensure that the fusion layer seamlessly adjusts to the feature space of the encoder, gradually enhancing the fusion performance. The proposed method can facilitate the comprehensive integration of feature vectors from both infrared and visible modalities, thus preserving the rich details inherent in each modal. MaeFuse not only introduces a novel perspective in the realm of fusion techniques but also stands out with impressive performance across various public datasets. The code is available at https://github.com/Henry-Lee-real/MaeFuse.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62471158, U23B2009)
This publication has 60 references indexed in Scilit:
- FusionBooster: A Unified Image Fusion Boosting ParadigmInternational Journal of Computer Vision, 2024
- Exploring Plain Vision Transformer Backbones for Object Detection2022
- Fusion from Decomposition: A Self-Supervised Decomposition Approach for Image Fusion2022
- ReCoNet: Recurrent Correction Network for Fast and Efficient Multi-modality Image Fusion2022
- CUFD: An encoder–decoder network for visible and infrared image fusion based on common and unique feature decompositionComputer Vision and Image Understanding, 2022
- RXDNFuse: A aggregated residual dense network for infrared and visible image fusionInformation Fusion, 2021
- IFCNN: A general image fusion framework based on convolutional neural networkInformation Fusion, 2020
- Infrared and visible image fusion via detail preserving adversarial learningInformation Fusion, 2020
- FusionGAN: A generative adversarial network for infrared and visible image fusionInformation Fusion, 2019
- Infrared and visible image fusion via gradient transfer and total variation minimizationInformation Fusion, 2016