Enhanced Multi-Stream Remote Sensing Spatiotemporal Fusion Network Based on Transformer and Dilated Convolution
Open Access
- 11 September 2022
- journal article
- research article
- Published by MDPI AG in Remote Sensing
- Vol. 14 (18) , 4544
Abstract
Remote sensing images with high temporal and spatial resolutions play a crucial role in land surface-change monitoring, vegetation monitoring, and natural disaster mapping. However, existing technical conditions and cost constraints make it very difficult to directly obtain remote sensing images with high temporal and spatial resolution. Consequently, spatiotemporal fusion technology for remote sensing images has attracted considerable attention. In recent years, deep learning-based fusion methods have been developed. In this study, to improve the accuracy and robustness of deep learning models and better extract the spatiotemporal information of remote sensing images, the existing multi-stream remote sensing spatiotemporal fusion network MSNet is improved using dilated convolution and an improved transformer encoder to develop an enhanced version called EMSNet. Dilated convolution is used to extract time information and reduce parameters. The improved transformer encoder is improved to further adapt to image-fusion technology and effectively extract spatiotemporal information. A new weight strategy is used for fusion that substantially improves the prediction accuracy of the model, image quality, and fusion effect. The superiority of the proposed approach is confirmed by comparing it with six representative spatiotemporal fusion algorithms on three disparate datasets. Compared with MSNet, EMSNet improved SSIM by 15.3% on the CIA dataset, ERGAS by 92.1% on the LGC dataset, and RMSE by 92.9% on the AHB dataset.Keywords
Funding Information
- National Natural Science Foundation of China (61972060, U1713213, 62027827, 2019YFE0110800, cstc2020jcyj-zdxmX0025, cstc2019cxcyljrc-td0270)
- National Key Research and Development Program of China (61972060, U1713213, 62027827, 2019YFE0110800, cstc2020jcyj-zdxmX0025, cstc2019cxcyljrc-td0270)
- Natural Science Foundation of Chongqing (61972060, U1713213, 62027827, 2019YFE0110800, cstc2020jcyj-zdxmX0025, cstc2019cxcyljrc-td0270)
This publication has 40 references indexed in Scilit:
- Assessing the accuracy of blending Landsat–MODIS surface reflectances in two landscapes with contrasting spatial and temporal dynamics: A framework for algorithm selectionRemote Sensing of Environment, 2013
- A review of large area monitoring of land cover change using Landsat dataRemote Sensing of Environment, 2012
- An enhanced spatial and temporal adaptive reflectance fusion model for complex heterogeneous regionsRemote Sensing of Environment, 2010
- Generation of dense time series synthetic Landsat data through data blending with MODIS using a spatial and temporal adaptive reflectance fusion modelRemote Sensing of Environment, 2009
- A new data fusion model for high spatial- and temporal-resolution mapping of forest disturbance based on Landsat and MODISRemote Sensing of Environment, 2009
- Real-time monitoring and short-term forecasting of land surface phenologyRemote Sensing of Environment, 2006
- Object-based Detailed Vegetation Classification with Airborne High Spatial Resolution Remote Sensing ImageryPhotogrammetric Engineering & Remote Sensing, 2006
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004
- Unmixing-based multisensor multiresolution image fusionIEEE Transactions on Geoscience and Remote Sensing, 1999
- Robust Estimation of a Location Parameter1992