Patrice Enrique Carbonneau

Recently, there has been significant progress in the area of semantic classification of water bodies at global scales with deep learning. For the key purposes of water inventory and change detection, advanced deep learning classifiers such as UNets and Vision Transformers have been shown to be both accurate and flexible when applied to large-scale, or even global, satellite image datasets from optical (e.g., Sentinel-2) and radar sensors (e.g., Sentinel-1). Most of this work is conducted with optical sensors, which usually have better image quality, but their obvious limitation is cloud cover, which is why radar imagery is an important complementary dataset. However, radar imagery is generally more sensitive to soil moisture than optical data. Furthermore, topography and wind-ripple effects can alter the reflected intensity of radar waves, which can induce errors in water classification models that fundamentally rely on the fact that water is darker than the surrounding landscape. In this paper, we develop a solution to the use of Sentinel-1 radar images for the semantic classification of water bodies that uses style transfer with multi-modal and multi-temporal image fusion. Instead of developing new semantic classification models that work directly on Sentinel-1 images, we develop a global style transfer model that produces synthetic Sentinel-2 images from Sentinel-1 input. The resulting synthetic Sentinel-2 imagery can then be classified with existing models. This has the advantage of obviating the need for large volumes of manually labeled Sentinel-1 water masks. Next, we show that fusing an 8-year cloud-free composite of the near-infrared band 8 of Sentinel-2 to the input Sentinel-1 image improves the classification performance. Style transfer models were trained and validated with global scale data covering the years 2017 to 2024, and include every month of the year. When tested against a global independent benchmark, S1S2-Water, the semantic classifications produced from our synthetic imagery show a marked improvement with the use of image fusion. When we use only Sentinel-1 data, we find an overall IoU (Intersection over Union) score of 0.70, but when we add image fusion, the overall IoU score rises to 0.93.

Rivers occupy less than 1% of the earth’s surface and yet they perform ecosystem service functions that are crucial to civilisation. Global monitoring of this asset is within reach thanks to the development of big data portals such as Google Earth Engine (GEE) but several challenges relating to output quality and processing efficiency remain. In this technical note, we present a new deep learning pipeline that uses attention-based deep learning to perform state-of-the-art semantic classification of fluvial landscapes with Sentinel-2 imagery accessed via GEE. We train, validate and test the network on a multi-seasonal and multi-annual dataset drawn from a study site that covers 89% of the Earth’s surface. F1-scores for independent test data not used in model training reach 92% for rivers and 96% for lakes. This is achieved without post-processing and significantly reduced computation times, thus making automated global monitoring of rivers achievable.

Machine learning algorithms applied on the publicly available Sentinel 2 images (S2) are opening the opportunity to automatically classify and monitor fluvial geomorphic feature (such as sediment bars or water channels) dynamics across scales. However, there are few analyses on the relative importance of S2 spatial versus temporal resolution in the context of geomorphic research. In a dynamic, braided reach of the Sesia River (Northern Italy), we thus analyzed how the inherent uncertainty associated with S2's spatial resolution (10 m pixel size) can impact the significance of the active channel (a combination of sediment and water) delineation, and how the S2's weekly temporal resolution can influence the interpretation of its evolutionary trajectory. A comparison with manually classified images at higher spatial resolutions (Planet: 3 m and orthophoto: 0.3 m) shows that the automatically classified water is ∼20% underestimated whereas sediments are ∼30% overestimated. These classification errors are smaller than the geomorphic changes detected in the 5 years analyzed, so the derived active channel trajectory can be considered robust. The comparison across resolutions also highlights that the yearly Planet‐ and S2‐derived active channel trajectory are analogous and they are both more effective in capturing the river geomorphic response after major flood events than the trajectory derived from sequential multiannual orthophotos. More analyses of this type, across different types of river could give insights on the transferability of the spatial uncertainty boundaries found as well as on the spatial and temporal resolution trade‐off needed for supporting different geomorphic analyses.

Recently, deep learning has been increasingly applied to global mapping of land‐use and land‐cover classes. However, very few studies have addressed the problem of separating lakes from rivers, and to our knowledge, none have addressed the issue of mapping fluvial sediment bars. We present the first global scale inventory of fluvial gravel bars. Our workflow is based on a state‐of‐the‐art fully convolutional neural network which is applied to Sentinel‐2 imagery at a resolution of 10 m. We use Google Earth Engine to access these data for a study site that covers 89% of the Earth's surface. We count 8.9 million gravel bars with an estimated area of 41 000 km². Crucially, the workflow we present can be executed within a month of highly automated processing and thus allows for global scale, monthly, monitoring of gravel bars and associated rivers.

Much progress has been made over the last decade to build global inventories of freshwater resources. However, existing freshwater inventories are generally produced for a fixed period in time and/or do not discriminate lakes and rivers. The emergence of deep learning methods and Big Data platforms such as Google Earth Engine offers a potential solution. Here we present a unique raster dataset of 6.7 terapixels at a spatial resolution of 10 m produced with a deep learning workflow. This dataset uses Sentinel-2 imagery downloaded from Google Earth Engine to delineate freshwater as separate semantic classes of rivers and lakes. Our study site covers the non-polar globe, 89% of the terrestrial landmass, with repeat surveys for the months of April, July and October 2021. This gives us the first global-scale direct count of lakes larger than 1 hectare as ~7.3 million. Also the repeat surveys allow us to present the first intra-annual ranges for the areas distinctly occupied by rivers and lakes where we find that basins influenced by the Asian summer monsoon (e.g. the Ganges) closely followed by the Amazon display the largest intra-annual range of river area per unit basin area. Finally, we find that whilst maximum of lake water surface occurs in October 2021, the maximum river surface is in July 2021 leading us to conclude that the global maximum occurrences of river water and lake water are not synchronous.

Much progress has been made over the last decade to build global inventories of freshwater resources. However, existing freshwater inventories are generally produced for a fixed period in time and/or do not discriminate lakes and rivers. The emergence of deep learning methods and Big Data platforms such as Google Earth Engine offers a potential solution. Here we presenta unique global raster dataset of 6.7 terapixels at a spatial resolution of 10 m produced with a deep learning workflow. This dataset uses Sentinel-2 imagery downloaded from Google Earth Engine to delineatefreshwater as separate semantic classes of rivers and lakes. Ourstudy site covers the non-polar globe, 89% of the terrestrial landmass, with repeat surveys for the months of April, July and October 2021. This gives us the first global-scale direct count of lakes larger than 1 hectare as ~7.3 million. Also the repeat surveys allow us topresent the firstintra-annual ranges for the areas distinctly occupied by rivers and lakes where we find that basins influenced by the Asian summer monsoon (e.g. the Ganges) closely followed by the Amazon display the largest intra-annual range of river area per unit basin area. Finally, we find that whilst maximum of lake water surface occurs in October 2021, the maximum river surfaceis in July 2021 leading us to conclude that the global maximum occurrences of river waterand lake water are not synchronous.

A comprehensive understanding of river dynamics requires the grain size distribution of bed sediments and its variation across different temporal and spatial scales. Several techniques are already available for grain size assessment based on field and remotely sensed data. However, the existing methods are only applicable on small spatial scales and on short time scales. Thus, the operational measurement of grain size distribution of riverbed sediments at the catchment scale remains an open problem. A solution could be the use of satellite images as the main imaging platform. However, this would entail retrieving information at sub‐pixel scales.In this study, we propose a new approach to retrieve sub‐pixel scale grain size class information from Copernicus Sentinel‐2 imagery building upon a new image‐based grain size mapping procedure. Three Italian gravel‐bed rivers featuring different morphologies were selected for unmanned aerial vehicle (UAV) acquisitions, field surveys and laboratory analysis meant to serve as ground truth grain size data, ranging from medium sand to coarse gravel. Grain size maps on the river bars were generated in each study site by exploiting image texture measurements, upscaled and co‐registered with Sentinel‐2 data resolution.Relationships between the grain sizes measured and the reflectance values in Sentinel‐2 imagery were analysed using a machine learning framework. Results show statistically significant predictive models (MAE of ±8.34 mm and R² = 0.92). The trained model was applied on 300 km of the Po River in Italy and allowed us to identify the gravel–sand transition occurring along this river length.Therefore, the approach presented here—based on freely available satellite data calibrated by low‐cost drone‐derived imagery—represents a promising step towards an automated surface mean grain size mapping over long river length, easily repeated through time for monitoring purposes.

Historical aerial imagery dating back to the mid-twentieth century offers high potential to distinguish anthropogenic impacts from natural causes of environmental change and reanalyze the long-term surface evolution from local to regional scales. However, the older portion of the imagery is often acquired in panchromatic grayscale thus making image classification a very challenging task.  This research aims to compare deep learning image colorisation methods, namely, , the Neural Style Transfer (NST) and the Cycle Generative Adversarial Network (CycleGAN), for colorizing archival images of Japanese river basins for land cover analysis. Historical monochrome images were examined with `4096 x 4096` pixels of three river basins, i.e., the Kurobe, Tenryu, and Chikugo Rivers. In the NST method, we used the transfer learning model with optimal hyperparameters that had already been fine-tuned for the river basin colorization of the archival river images (Ishii et al., 2021). As for the CycleGAN method, we trained the CycleGAN with 8000 image tiles of `256 x256` pixels to obtain the optimal hyperparameters for the river basin colorization. The image tiles used in training consisted of 10 land-use types, including paddy fields, agricultural lands, forests, wastelands, cities and villages, transportation land, rivers, lakes, coastal areas, and so forth. The training result of the CycleGAN reached an optimal model in which the root mean square error (RMSE) of colorization was 18.3 in 8-bit RGB color resolution with optimal hyperparameters of the dropout ratio (0.4), cycle consistency loss (10), and identity mapping loss (0.5). Colorization comparison of the two-deep learning methods gave us the following three findings. (i) CycleGAN requires much less training effort than the NST because the CycleGAN used an unsupervised learning algorithm. CycleGAN used 8000 images without labelling for training while the NST used 60k with labelling in transfer learning. (ii) The colorization quality of the two methods was basically the same in the evaluation stage; RMSEs in CycleGAN were 15.4 for Kurobe, 13.7 for Tenryu and 18.7 for Chikugo, while RMSE in NST were 9.9 for Kurobe, 15.8 for Tenryu, and 14.2 for Chikugo, respectively. (iii) The CycleGAN indicated much higher performance on the colorization of dull surfaces without any textual features, such as the river course in Tenryu River, than the NST. In future research work, colorized imagery by both the NST and CycleGAN will be further used for land cover classification with AI technology to investigate its role in image recognition. [Reference]: Ishii, R. et al.(2021) Colorization of archival aerial imagery using deep learning, EGU General Assembly 2021, EGU21-11925, https://doi.org/10.5194/egusphere-egu21-11925.

A wealth of research has focused on elucidating the key controls on mass loss from the Greenland and Antarctic ice sheets in response to climate forcing, specifically in relation to the drivers of marine-terminating outlet glacier change. The manual methods traditionally used to monitor change in satellite imagery of marine-terminating outlet glaciers are time-consuming and can be subjective, especially where mélange exists at the terminus. Recent advances in deep learning applied to image processing have created a new frontier in the field of automated delineation of glacier calving fronts. However, there remains a paucity of research on the use of deep learning for pixel-level semantic image classification of outlet glacier environments. Here, we apply and test a two-phase deep learning approach based on a well-established convolutional neural network (CNN) for automated classification of Sentinel-2 satellite imagery. The novel workflow, termed CNN-Supervised Classification (CSC) is adapted to produce multi-class outputs for unseen test imagery of glacial environments containing marine-terminating outlet glaciers in Greenland. Different CNN input parameters and training techniques are tested, with overall F1 scores for resulting classifications reaching up to 94 % for in-sample test data (Helheim Glacier) and 96 % for out-of-sample test data (Jakobshavn Isbrae and Store Glacier), establishing a state of the art in classification of marine-terminating glaciers in Greenland. Predicted calving fronts derived using optimal CSC input parameters have a mean deviation of 56.17 m (5.6 px) and median deviation of 24.7 m (2.5 px) from manually digitised fronts. This demonstrates the transferability and robustness of the deep learning workflow despite complex and seasonally variable imagery. Future research could focus on the integration of deep learning classification workflows with free cloud-based platforms, to efficiently classify imagery and produce datasets for a range of glacial applications without the need for substantial prior experience in coding or deep learning.

Archival imagery dating back to the mid-twentieth century holds information that pre-dates urban expansion and the worst impacts of climate change.  In this research, we examine deep learning colorisation methods applied to historical aerial images in Japan.  Specifically, we attempt to colorize monochrome images of river basins by applying the method of Neural Style Transfer (NST).    First, we created RGB orthomosaics (1m) for reaches of 3 Japanese rivers, the Kurobe, Ishikari, and Kinu rivers.  From the orthomosaics, we extract 60 thousand image tiles of `100 x100` pixels in order to train the CNN used in NST.  The Image tiles were classified into 6 classes: urban, river, forest, tree, grass, and paddy field.  Second, we use the VGG16 model pre-trained on ImageNet data in a transfer learning approach where we freeze a variable number of layers.  We fine-tuned the training epochs, learning rate, and frozen layers in VGG16 in order to derive the optimal CNN used in NST.  The fine tuning resulted in the F-measure accuracy of 0.961, 0.947, and 0.917 for the freeze layer in 7,11,15, respectively.  Third, we colorize monochrome aerial images by the NST with the retrained model weights.  Here used RGB images for 7 Japanese rivers and the corresponding grayscale versions to evaluate the present NST colorization performance.  The RMSE between the RGB and resultant colorized images showed the best performance with the model parameters of lower content layer (6), shallower freeze layer (7), and larger style/content weighting ratio (1.0 x10⁵).  The NST hyperparameter analysis indicated that the colorized images became rougher when the content layer selected deeper in the VGG model.  This is because the deeper the layer, the more features were extracted from the original image.  It was also confirmed that the Kurobe and Ishikari rivers indicated higher accuracy in colorisation.  It might come from the fact that the training dataset of the fine tuning was extracted from these river images.  Finally, we colorized historical monochrome images of Kurobe river with the best NST parameters, resulting in quality high enough compared with the RGB images.  The result indicated that the fine tuning of the NST model could achieve high performance to proceed further land cover classification in future research work.