A Generative Adversarial Network for Infrared and Visible Image Fusion Based on Semantic Segmentation
Open Access
Abstract
This paper proposes a new generative adversarial network for infrared and visible image fusion based on semantic segmentation (SSGAN), which can consider not only the low-level features of infrared and visible images, but also the high-level semantic information. Source images can be divided into foregrounds and backgrounds by semantic masks. The generator with a dual-encoder-single-decoder framework is used to extract the feature of foregrounds and backgrounds by different encoder paths. Moreover, the discriminator’s input image is designed based on semantic segmentation, which is obtained by combining the foregrounds of the infrared images with the backgrounds of the visible images. Consequently, the prominence of thermal targets in the infrared images and texture details in the visible images can be preserved in the fused images simultaneously. Qualitative and quantitative experiments on publicly available datasets demonstrate that the proposed approach can significantly outperform the state-of-the-art methods.Keywords
Funding Information
- National Natural Science Foundation of China (61771353, 61773295, 62072350, 41501505)
- Hubei Technology Innovation Project (2019AAA045)
- Innovation Fund for Industry, Education and Research in Chinese Universities-New Generation Information Technology Innovation Project (2019ITA02014)
- the Key Scientific and Technological Research Project of Hubei Provincial Education Department (D20201507)
This publication has 50 references indexed in Scilit:
- Novel fusion method for visible light and infrared images based on NSST–SF–PCNNInfrared Physics & Technology, 2014
- Infrared image enhancement through saliency feature analysis based on multi-scale decompositionInfrared Physics & Technology, 2014
- A new image fusion performance metric based on visual information fidelityInformation Fusion, 2013
- Assessment of image fusion procedures using entropy, image quality, and multispectral classificationJournal of Applied Remote Sensing, 2008
- Pixel- and region-based image fusion with complex waveletsInformation Fusion, 2007
- A wavelet-based image fusion tutorialPattern Recognition, 2004
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004
- Information measure for performance of image fusionElectronics Letters, 2002
- A categorization of multiscale-decomposition-based image fusion schemes with a performance study for a digital camera applicationProceedings of the IEEE, 1999
- In-fibre Bragg grating sensorsMeasurement Science and Technology, 1997