FuseVis: Interpreting Neural Networks for Image Fusion Using Per-Pixel Saliency Visualization
Open Access
Abstract
Image fusion helps in merging two or more images to construct a more informative single fused image. Recently, unsupervised learning-based convolutional neural networks (CNN) have been used for different types of image-fusion tasks such as medical image fusion, infrared-visible image fusion for autonomous driving as well as multi-focus and multi-exposure image fusion for satellite imagery. However, it is challenging to analyze the reliability of these CNNs for the image-fusion tasks since no groundtruth is available. This led to the use of a wide variety of model architectures and optimization functions yielding quite different fusion results. Additionally, due to the highly opaque nature of such neural networks, it is difficult to explain the internal mechanics behind its fusion results. To overcome these challenges, we present a novel real-time visualization tool, named FuseVis, with which the end-user can compute per-pixel saliency maps that examine the influence of the input image pixels on each pixel of the fused image. We trained several image fusion-based CNNs on medical image pairs and then using our FuseVis tool we performed case studies on a specific clinical application by interpreting the saliency maps from each of the fusion methods. We specifically visualized the relative influence of each input image on the predictions of the fused image and showed that some of the evaluated image-fusion methods are better suited for the specific clinical application. To the best of our knowledge, currently, there is no approach for visual analysis of neural networks for image fusion. Therefore, this work opens a new research direction to improve the interpretability of deep fusion networks. The FuseVis tool can also be adapted in other deep neural network-based image processing applications to make them interpretable.Keywords
All Related Versions
This publication has 60 references indexed in Scilit:
- The multiscale directional bilateral filter and its application to multisensor image fusionInformation Fusion, 2012
- Pixel-level image fusion with simultaneous orthogonal matching pursuitInformation Fusion, 2012
- A non-reference image fusion metric based on mutual information of image featuresComputers and Electrical Engineering, 2011
- A novel algorithm of image fusion using shearletsOptics Communications, 2011
- Image fusion based on a new contourlet packetInformation Fusion, 2010
- Multimodality medical image fusion based on multiscale geometric analysis of contourlet transformNeurocomputing, 2008
- The Alzheimer's disease neuroimaging initiative (ADNI): MRI methodsJournal of Magnetic Resonance Imaging, 2008
- Pixel- and region-based image fusion with complex waveletsInformation Fusion, 2007
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004
- A general framework for multiresolution image fusion: from pixels to regionsInformation Fusion, 2003