Applying self-supervised learning for semantic cloud segmentation of all-sky images
Authors:
- Yann FabelYFYann Fabel
- Bijan NouriBNBijan Nouri
- Niklas BlumNBNiklas Blum
- Marcel HasenbalgMHMarcel Hasenbalg
- Luis F. ZarzalejoLZLuis F. Zarzalejo
Open Access
- 14 February 2022
- journal article
- research article
- Published by Copernicus GmbH in Atmospheric Measurement Techniques
- Vol. 15 (3) , 797-809
Abstract
Semantic segmentation of ground-based all-sky images (ASIs) can provide high-resolution cloud coverage information of distinct cloud types, applicable for meteorology-, climatology- and solar-energy-related applications. Since the shape and appearance of clouds is variable, and there is high similarity between cloud types, a clear classification is difficult. Therefore, most state-of-the-art methods focus on the distinction between cloudy and cloud-free pixels without taking into account the cloud type. On the other hand, cloud classification is typically determined separately at the image level, neglecting the cloud's position and only considering the prevailing cloud type. Deep neural networks have proven to be very effective and robust for segmentation tasks; however they require large training datasets to learn complex visual features. In this work, we present a self-supervised learning approach to exploit many more data than in purely supervised training and thus increase the model's performance. In the first step, we use about 300 000 ASIs in two different pretext tasks for pretraining. One of them pursues an image reconstruction approach. The other one is based on the DeepCluster model, an iterative procedure of clustering and classifying the neural network output. In the second step, our model is fine-tuned on a small labeled dataset of 770 ASIs, of which 616 are used for training and 154 for validation. For each of them, a ground truth mask was created that classifies each pixel into clear sky or a low-layer, mid-layer or high-layer cloud. To analyze the effectiveness of self-supervised pretraining, we compare our approach to randomly initialized and pretrained ImageNet weights using the same training and validation sets. Achieving 85.8 % pixel accuracy on average, our best self-supervised model outperforms the conventional approaches of random (78.3 %) and pretrained ImageNet initialization (82.1 %). The benefits become even more evident when regarding precision, recall and intersection over union (IoU) of the respective cloud classes, where the improvement is between 5 and 20 percentage points. Furthermore, we compare the performance of our best model with regards to binary segmentation with a clear-sky library (CSL) from the literature. Our model outperforms the CSL by over 7 percentage points, reaching a pixel accuracy of 95 %.Keywords
All Related Versions
Funding Information
- Bundesministerium für Wirtschaft und Energie (0324307A)
- Horizon 2020
This publication has 52 references indexed in Scilit:
- A method for cloud detection and opacity classification based on ground based sky imageryAtmospheric Measurement Techniques, 2012
- Cloud detection and classification with the use of whole-sky ground-based imagesAtmospheric Research, 2012
- Intra-hour forecasting with a total sky imager at the UC San Diego solar energy testbedSolar Energy, 2011
- A Hybrid Thresholding Algorithm for Cloud Detection on Ground-Based Color ImagesJournal of Atmospheric and Oceanic Technology, 2011
- Automatic cloud classification of whole sky imagesAtmospheric Measurement Techniques, 2010
- On the twilight zone between clouds and aerosolsGeophysical Research Letters, 2007
- Retrieving Cloud Characteristics from Ground-Based Daytime Color All-Sky ImagesJournal of Atmospheric and Oceanic Technology, 2006
- A Simple Method for the Assessment of the Cloud Cover State in High-Latitude Regions by a Ground-Based Digital CameraJournal of Atmospheric and Oceanic Technology, 2006
- A new airmass independent formulation for the Linke turbidity coefficientSolar Energy, 2002
- Calculation of surface and top of atmosphere radiative fluxes from physical quantities based on ISCCP data sets: 2. Validation and first resultsJournal of Geophysical Research: Atmospheres, 1995