LDRepFM: A Real-Time End-to-End Visible and Infrared Image Fusion Model Based on Layer Decomposition and Re-Parameterization
- 29 May 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 72 (00189456) , 1-12
Abstract
Visible (VIS) and infrared (IR) image fusion (VIF) is a technique used to synthesize the fused image of high visual perception. Existing fusion methods typically work by discovering the commons underlying the two modalities and fusing them in the common space. However, these methods often ignore the modality differences such as fuzzy details in the IR image and their well-designed architectures also lead to slow fusion speed. To address these issues, we propose a real-time end-to-end VIF model based on layer decomposition and re-parameterization (LDRepFM). This model is composed of a Layer Decomposition Guidance Network (LDGNet) and a Re-parameterization Fusion Network (RepFNet). Firstly, the LDGNet is used to alleviate the visual quality degradation of the fused image by decomposing the IR image into structural layer and fuzzy layer. Secondly, in order to achieve a favorable trade-off between the fusion speed and evaluation metrics, the RepFNet is utilized to decouple the training-time multi-branch and inference-time plain architecture. Thirdly, the structural layer that has been decomposed by LDGNet is utilized in constructing the guidance fusion loss, which is aimed at optimizing RepFNet. Finally, experiments conducted on the publicly available TNO, RoadScene, M3FD, and RegDB datasets demonstrate the performance of the proposed method to be comparable to the state-of-the-art in terms of both visual effect and quantitative metrics. The code is publically available at https://github.com/luming1314/LDRepFM.Keywords
Funding Information
- Fundamental Research Funds for the Central Universities (JUSRP41908)
- National Natural Science Foundation of China (61362030, 61201429)
- China Postdoctoral Science Foundation (2015M581720, 2016M600360)
This publication has 53 references indexed in Scilit:
- Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: A reviewInformation Fusion, 2018
- The TNO Multiband Image Data CollectionData in Brief, 2017
- Infrared and visible image fusion based on visual saliency map and weighted least square optimizationInfrared Physics & Technology, 2017
- Person Recognition System Based on a Combination of Body Images from Visible Light and Thermal CamerasSensors, 2017
- Infrared and visible image fusion via gradient transfer and total variation minimizationInformation Fusion, 2016
- Infrared and visible images fusion based on RPCA and NSCTInfrared Physics & Technology, 2016
- Deep Residual Learning for Image Recognition2016
- You Only Look Once: Unified, Real-Time Object Detection2016
- Image fusion based on non-negative matrix factorization and infrared feature extraction2013
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004