A Cross-Scale Iterative Attentional Adversarial Fusion Network for Infrared and Visible Images
- 25 January 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 33 (8) , 3677-3688
Abstract
Recent existing methods generally adopt a simple concatenation or addition strategy to integrate features at the fusion layer, failing to adequately consider the intrinsic characteristics of different modal images and feature interaction of different scales, which may produce a limited fusion performance. Toward this end, we introduce a cross-scale iterative attentional adversarial fusion network, namely CrossFuse. More specifically, in the generator, we design a cross-modal attention integrated module to merge the intrinsic content of different modal images. The parallel spatial-independent and channel-independent pathways are proposed to calculate the attentional weights, which are assigned to measure the activity levels of source images at the same scale. Moreover, we construct a cross-scale iterative decoder framework to interact with different modality features at different scales, which can constantly optimize their activity levels. By this means, the generator learns to integrate their modality characteristics via attentional weights in an iterative manner, and the generated result characterizes competitive infrared radiant intensity and distinct visible detail description. Extensive experiments on three different benchmarks demonstrate that our CrossFuse outperforms other nine state-of-the-art methods in terms of fusion performance, generalization ability and computational efficiency. Our codes will be released at https://github.com/Zhishe-Wang/CrossFuse.Keywords
Funding Information
- Open Foundation of Shanxi Key Laboratory of Signal Capturing and Processing (ISPT2020-4)
- Outstanding Graduate Student Innovation Program of Taiyuan University of Science and Technology (SY2022079)
- Natural Science Foundation of Zhejiang Province (LY23F030001)
This publication has 42 references indexed in Scilit:
- FusionGAN: A generative adversarial network for infrared and visible image fusionInformation Fusion, 2019
- Multi-sensor image enhanced fusion algorithm based on NSST and top-hat transformationOptik, 2015
- Adaptive fusion method of visible light and infrared images based on non-subsampled shearlet transform and fast non-negative matrix factorizationInfrared Physics & Technology, 2014
- Fast-FMI: Non-reference image fusion metric2014
- A new image fusion performance metric based on visual information fidelityInformation Fusion, 2013
- Objective Assessment of Multiresolution Image Fusion Algorithms for Context Enhancement in Night Vision: A Comparative StudyIEEE Transactions on Pattern Analysis and Machine Intelligence, 2011
- Electromagnetic modeling of near-field phase-shifting contact lithography with broadband ultraviolet illuminationOptik, 2005
- Information measure for performance of image fusionElectronics Letters, 2002
- In-fibre Bragg grating sensorsMeasurement Science and Technology, 1997
- Image quality measures and their performanceIEEE Transactions on Communications, 1995