LPGAN: A LBP-based Proportional Input Generative Adversarial Network for Image Fusion
Preprint
- 6 April 2023
- preprint
- Published by MDPI AG in Preprints.org
Abstract
Image fusion is the process of combining multiple input images from single or multiple imaging modalities into a fused image, which is expected to be more informative for human or machine perception as compared to any of the input images. In this paper, we propose a novel method based on deep learning for fusing infrared images and visible images, named the LBP-based proportional input generative adversarial network (LPGAN). In the image fusion task, the preservation of structural similarity and image gradient information is contradictory, and it is difficult for both to achieve good performance at the same time. To solve this problem, we innovatively introduce Local Binary Patterns (LBP) into Generative Adversarial Networks (GANs), which effectively utilize the texture features of the source images, so that the network has stronger feature extraction ability and anti-interference ability. In the feature extraction stage, we introduce a pseudo-siamese network for the generator to extract the detailed features and the contrast features. At the same time, considering the characteristic distribution of different modal images, we propose a 1:4 scale input mode. Extensive experiments on the publicly available TNO dataset and CVC14 dataset show that the proposed method achieves the state-of-the-art performance. We also test the universality of LPGAN through the fusion of RGB and infrared images on the RoadScene dataset. In addition, LPGAN is applied to multi-spectral remote sensing image fusion. Both qualitative and quantitative experiments demonstrate that our LPGAN can not only achieve good structural similarity, but also retain rich detailed information.Keywords
All Related Versions
- Version 1, 2023-04-06, Preprints
- Published version: Remote Sensing, 15 (9), 2440.
This publication has 54 references indexed in Scilit:
- Infrared and visible image fusion methods and applications: A surveyInformation Fusion, 2019
- Deep learning for pixel-level image fusion: Recent advances and future prospectsInformation Fusion, 2018
- Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: A reviewInformation Fusion, 2018
- Multi-focus image fusion with a deep convolutional neural networkInformation Fusion, 2017
- Pixel-level image fusion: A survey of the state of the artInformation Fusion, 2017
- A review of remote sensing image fusion methodsInformation Fusion, 2016
- Infrared and visible image fusion via gradient transfer and total variation minimizationInformation Fusion, 2016
- Medical image fusion: A survey of the state of the artInformation Fusion, 2014
- Hybrid DDCT-PCA based multi sensor image fusionJournal of Optics, 2013
- A new image fusion performance metric based on visual information fidelityInformation Fusion, 2013