Applying self-supervised learning for semantic cloud segmentation of all-sky images
Preprint
- 19 March 2021
- preprint
- Published by Copernicus GmbH in EGUsphere
- Vol. 2021, 1-20
Abstract
Semantic segmentation of ground-based all-sky images (ASIs) can provide high-resolution cloud coverage information of distinct cloud types, applicable for meteorology, climatology and solar energy-related applications. Since the shape and appearance of clouds is variable and there is high similarity between cloud types, a clear classification is difficult. Therefore, most state-of-the-art methods focus on the distinction between cloudy- and cloudfree-pixels, without taking into account the cloud type. On the other hand, cloud classification is typically determined separately on image-level, neglecting the cloud's position and only considering the prevailing cloud type. Deep neural networks have proven to be very effective and robust for segmentation tasks, however they require large training datasets to learn complex visual features. In this work, we present a self-supervised learning approach to exploit much more data than in purely supervised training and thus increase the model's performance. In the first step, we use about 300,000 ASIs in two different pretext tasks for pretraining. One of them pursues an image reconstruction approach. The other one is based on the DeepCluster model, an iterative procedure of clustering and classifying the neural network output. In the second step, our model is fine-tuned on a small labeled dataset of 770 ASIs, of which 616 are used for training and 154 for validation. For each of them, a ground truth mask was created that classifies each pixel into clear sky, low-layer, mid-layer or high-layer cloud. To analyze the effectiveness of self-supervised pretraining, we compare our approach to randomly initialized and pretrained ImageNet weights, using the same training and validation sets. Achieving 85.8 % pixel-accuracy on average, our best self-supervised model outperforms the conventional approaches of random (78.3 %) and pretrained ImageNet initialization (82.1 %). The benefits become even more evident when regarding precision, recall and intersection over union (IoU) on the respective cloud classes, where the improvement is between 5 and 20 % points. Furthermore, we compare the performance of our best model on binary segmentation with a clear-sky library (CSL) from the literature. Our model outperforms the CSL by over 7 % points, reaching a pixel-accuracy of 95 %.Keywords
All Related Versions
- Published version: Atmospheric Measurement Techniques, 15 (3), 797.
Funding Information
- Bundesministerium für Wirtschaft und Energie (0324307A)
- Horizon 2020
This publication has 27 references indexed in Scilit:
- Benchmarking of six cloud segmentation algorithms for ground-based all-sky imagersSolar Energy, 2020
- An Efficient Solution for Semantic Segmentation of Three Ground‐based Cloud DatasetsEarth and Space Science, 2020
- CloudNet: Ground‐Based Cloud Classification With Deep Convolutional Neural NetworkGeophysical Research Letters, 2018
- Validation of an all‐sky imager–based nowcasting system for industrial PV plantsProgress in Photovoltaics: Research and Applications, 2017
- The thin border between cloud and aerosol: Sensitivity of several ground based observation techniquesAtmospheric Research, 2017
- Short-term forecasting of high resolution local DNI maps with multiple fish-eye cameras in stereoscopic modeAIP Conference Proceedings, 2017
- Short-term irradiance forecasting using skycams: Motivation and developmentSolar Energy, 2014
- Cloud detection and classification with the use of whole-sky ground-based imagesAtmospheric Research, 2012
- Intra-hour forecasting with a total sky imager at the UC San Diego solar energy testbedSolar Energy, 2011
- On the twilight zone between clouds and aerosolsGeophysical Research Letters, 2007