Deep Adaptive Pansharpening via Uncertainty-Aware Image Fusion
- 21 April 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 61 (01962892) , 1-15
Abstract
Pansharpening is a procedure that fuses high-resolution panchromatic (PAN) images and low-resolution multispectral (LMS) images to derive high-resolution multispectral (HMS) images. Despite its rapid development, most existing pansharpening techniques integrate the information of PAN and LMS invariantly in the spatial dimension, ignoring the uneven spatial dependence of restoring HMS with the aid of PAN information and resulting in ineffective fusion results. In this work, we propose an Uncertainty-aware Adaptive Pansharpening Network (UAPN) that integrates PAN information spatial-variantly to restore LMS information with an uncertainty mechanism. Specifically, we first estimate the epistemic and aleatoric uncertainties together, which model the spatial-variant distributions of restoring the LMS image to the HMS image. Then, we introduce Uncertainty-conditioned Adaptive Convolution (UAC) to adaptively integrate LMS and PAN information, where its parameters are spatially variable by conditioning on the uncertainty estimations. Furthermore, we propose a multi-stage uncertainty-driven loss function to explicitly force the network to concentrate on restoring challenging areas of the LMS image. Extensive experimental results demonstrate the superiority of our UAPN with fewer parameters and flops, outperforming other state-of-the-art methods both qualitatively and quantitatively on multiple satellite datasets. The code is available at https://github.com/keviner1/UAPN..Keywords
Funding Information
- JKW Research Funds (20-163-14-LZ-001-004-01)
- Anhui Provincial Natural Science Foundation (2108085UD12)
This publication has 54 references indexed in Scilit:
- Spatial-Hessian-Feature-Guided Variational Model for Pan-SharpeningIEEE Transactions on Geoscience and Remote Sensing, 2015
- SIRF: Simultaneous Satellite Image Registration and Fusion in a Unified FrameworkIEEE Transactions on Image Processing, 2015
- Contrast and Error-Based Fusion Schemes for Multispectral Image PansharpeningIEEE Geoscience and Remote Sensing Letters, 2013
- A Variational Approach for Pan-SharpeningIEEE Transactions on Image Processing, 2013
- Panchromatic sharpening of remote sensing images using a multiscale Kalman filterPattern Recognition, 2007
- Improving Component Substitution Pansharpening Through Multivariate Regression of MS $+$Pan DataIEEE Transactions on Geoscience and Remote Sensing, 2007
- A Variational Model for P+XS Image FusionInternational Journal of Computer Vision, 2006
- Smoothing Filter-based Intensity Modulation: A spectral preserve image fusion technique for improving spatial detailsInternational Journal of Remote Sensing, 2000
- Multisensor Image Fusion Using the Wavelet TransformGraphical Models and Image Processing, 1995
- Color enhancement of highly correlated images. II. Channel ratio and “chromaticity” transformation techniquesRemote Sensing of Environment, 1987