A Robust Hybrid Deep Learning Model for Spatiotemporal Image Fusion
Open Access
- 9 December 2021
- journal article
- research article
- Published by MDPI AG in Remote Sensing
- Vol. 13 (24) , 5005
Abstract
Dense time-series remote sensing data with detailed spatial information are highly desired for the monitoring of dynamic earth systems. Due to the sensor tradeoff, most remote sensing systems cannot provide images with both high spatial and temporal resolutions. Spatiotemporal image fusion models provide a feasible solution to generate such a type of satellite imagery, yet existing fusion methods are limited in predicting rapid and/or transient phenological changes. Additionally, a systematic approach to assessing and understanding how varying levels of temporal phenological changes affect fusion results is lacking in spatiotemporal fusion research. The objective of this study is to develop an innovative hybrid deep learning model that can effectively and robustly fuse the satellite imagery of various spatial and temporal resolutions. The proposed model integrates two types of network models: super-resolution convolutional neural network (SRCNN) and long short-term memory (LSTM). SRCNN can enhance the coarse images by restoring degraded spatial details, while LSTM can learn and extract the temporal changing patterns from the time-series images. To systematically assess the effects of varying levels of phenological changes, we identify image phenological transition dates and design three temporal phenological change scenarios representing rapid, moderate, and minimal phenological changes. The hybrid deep learning model, alongside three benchmark fusion models, is assessed in different scenarios of phenological changes. Results indicate the hybrid deep learning model yields significantly better results when rapid or moderate phenological changes are present. It holds great potential in generating high-quality time-series datasets of both high spatial and temporal resolutions, which can further benefit terrestrial system dynamic studies. The innovative approach to understanding phenological changes’ effect will help us better comprehend the strengths and weaknesses of current and future fusion models.Keywords
Funding Information
- National Science Foundation (1849821, 1951657)
- U.S. Department of Agriculture (2021-67021-33446)
- National Aeronautics and Space Administration (80NSSC21K0946)
This publication has 41 references indexed in Scilit:
- Comparison of Spatiotemporal Fusion Models: A ReviewRemote Sensing, 2015
- Multitemporal fusion of Landsat/TM and ENVISAT/MERIS for crop monitoringInternational Journal of Applied Earth Observation and Geoinformation, 2013
- Remote Sensing Based Detection of Crop Phenology for Agricultural Zones in China Using a New Threshold MethodRemote Sensing, 2013
- Unified fusion of remote-sensing imagery: generating simultaneously high-resolution synthetic spatial–temporal–spectral earth observationsRemote Sensing Letters, 2013
- Monitoring US agriculture: the US Department of Agriculture, National Agricultural Statistics Service, Cropland Data Layer ProgramGeocarto International, 2011
- An enhanced spatial and temporal adaptive reflectance fusion model for complex heterogeneous regionsRemote Sensing of Environment, 2010
- A new data fusion model for high spatial- and temporal-resolution mapping of forest disturbance based on Landsat and MODISRemote Sensing of Environment, 2009
- Monitoring vegetation phenology using MODISRemote Sensing of Environment, 2003
- Unmixing-based multisensor multiresolution image fusionIEEE Transactions on Geoscience and Remote Sensing, 1999
- Long Short-Term MemoryNeural Computation, 1997