Task-Generalized Adaptive Cross-Domain Learning for Multimodal Image Fusion
Authors:
- Mengyu WangMengyu Wang
- Zhenyu LiuZLZhenyu Liu
- Kun LiKLKun Li
- Yu WangYWYu Wang
- Yuwei WangYuwei Wang
- Yanyan WeiYWYanyan Wei
- Fei WangFWFei Wang
Preprint
Abstract
Multimodal Image Fusion (MMIF) aims to integrate complementary information from different imaging modalities to overcome the limitations of individual sensors. It enhances image quality and facilitates downstream applications such as remote sensing, medical diagnostics, and robotics. Despite significant advancements, current MMIF methods still face challenges such as modality misalignment, high-frequency detail destruction, and task-specific limitations. To address these challenges, we propose AdaSFFuse, a novel framework for task-generalized MMIF through adaptive cross-domain co-fusion learning. AdaSFFuse introduces two key innovations: the Adaptive Approximate Wavelet Transform (AdaWAT) for frequency decoupling, and the Spatial-Frequency Mamba Blocks for efficient multimodal fusion. AdaWAT adaptively separates the high- and low-frequency components of multimodal images from different scenes, enabling fine-grained extraction and alignment of distinct frequency characteristics for each modality. The Spatial-Frequency Mamba Blocks facilitate cross-domain fusion in both spatial and frequency domains, enhancing this process. These blocks dynamically adjust through learnable mappings to ensure robust fusion across diverse modalities. By combining these components, AdaSFFuse improves the alignment and integration of multimodal features, reduces frequency loss, and preserves critical details. Extensive experiments on four MMIF tasks -- Infrared-Visible Image Fusion (IVF), Multi-Focus Image Fusion (MFF), Multi-Exposure Image Fusion (MEF), and Medical Image Fusion (MIF) -- demonstrate AdaSFFuse's superior fusion performance, ensuring both low computational cost and a compact network, offering a strong balance between performance and efficiency. The code will be publicly available at https://github.com/Zhen-yu-Liu/AdaSFFuse.Keywords
All Related Versions
This publication has 57 references indexed in Scilit:
- Improving long-tailed pest classification using diffusion model-based data augmentationComputers and Electronics in Agriculture, 2025
- Low-light wheat image enhancement using an explicit inter-channel sparse transformerComputers and Electronics in Agriculture, 2024
- CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image FusionInternational Journal of Computer Vision, 2023
- ViGT: proposal-free video grounding with a learnable token in the transformerScience China Information Sciences, 2023
- Medical image fusion based on convolutional neural networks and non-subsampled contourlet transformExpert Systems with Applications, 2021
- Multi-focus image fusion with a deep convolutional neural networkInformation Fusion, 2017
- Multi-focus image fusion using dictionary-based sparse representationInformation Fusion, 2015
- A general framework for image fusion based on multi-scale transform and sparse representationInformation Fusion, 2015
- A new image fusion performance metric based on visual information fidelityInformation Fusion, 2013
- A non-reference image fusion metric based on mutual information of image featuresComputers and Electrical Engineering, 2011