Semantic Region Adaptive Fusion of Infrared and Visible Images via Dual-DeepLab Guidance
- 6 October 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 72 (00189456) , 1-16
Abstract
Given the potential diminishment of semantic information caused by previous non-region-specific maximum or weighted intensity losses under different lighting conditions, we propose a novel high-level semantic perception fusion framework, termed Semantic Region Adaptive Fusion (SRAFusion). We establish a prioritized preservation scheme for high-level semantic information (HLSI), gradient information, and intensity information, ranked in descending order of priority. Based on this prioritization scheme and prior knowledge of semantic distribution in the source images, we construct a ground-truth for the fusion task. Specifically, we capture the HLSI distribution of the source images by utilizing two independent semantic segmentation networks. Subsequently, we introduce Semantic Region Decision Blocks (SRDB) to partition the original scene into region with bimodal HLSI, region with unimodal HLSI, and region lacking HLSI. We then design specific loss functions to constrain the aforementioned regions, facilitating the integration of complete semantic information. Furthermore, taking into account the susceptibility of the visible segmentation network to lighting conditions, we employ a two-stage training strategy involving coarse-tuning and fine-tuning. This method aims to optimize one-stage training strategy and achieve a more accurate region delineation. Finally, qualitative and quantitative experiments conducted on publicly available datasets such as MFNet , RoadScene , and TNO demonstrate the superiority of our SRAFusion over state-of-the-art methods. Our code will be available: https://github.com/WenziCao/SRAFusion.Keywords
This publication has 63 references indexed in Scilit:
- Infrared and visible image fusion via gradient transfer and total variation minimizationInformation Fusion, 2016
- Fast R-CNN2015
- Infrared and visible image fusion with the use of multi-scale edge-preserving decomposition and guided image filterInfrared Physics & Technology, 2015
- Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decompositionOptics Communications, 2015
- No-reference image quality assessment based on spatial and spectral entropiesSignal Processing: Image Communication, 2014
- Fast saliency-aware multi-modality image fusionNeurocomputing, 2013
- A new image fusion performance metric based on visual information fidelityInformation Fusion, 2013
- Visual attention detection in video sequences using spatiotemporal cues2006
- Information measure for performance of image fusionElectronics Letters, 2002
- Image quality measures and their performanceIEEE Transactions on Communications, 1995