TransCloudSeg: Ground-Based Cloud Image Segmentation With Transformer
Open Access
- 27 July 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
- Vol. 15 (19391404) , 6121-6132
Abstract
Cloud image segmentation plays an important role in ground-based cloud observation. Recently, most existing methods for ground-based cloud image segmentation learn feature representations using the convolutional neural network (CNN), which results in the loss of global information because of the limited receptive field size of the filters in the CNN. In this article, we propose a novel deep model named TransCloudSeg, which makes full use of the advantages of the CNN and transformer to extract detailed information and global contextual information for ground-based cloud image segmentation. Specifically, TransCloudSeg hybridizes the CNN and transformer as the encoders to obtain different features. To recover and fuse the feature maps from the encoders, we design the CNN decoder and the transformer decoder for TransCloudSeg. After obtaining two sets of feature maps from two different decoders, we propose the heterogeneous fusion module to effectively fuse the heterogeneous feature maps by applying the self-attention mechanism. We conduct a series of experiments on Tianjin Normal University large-scale cloud detection database and Tianjin Normal University cloud detection database, and the results show that our method achieves a better performance than other state-of-the-art methods, thus proving the effectiveness of the proposed TransCloudSeg.Keywords
Funding Information
- National Natural Science Foundation of China (62171321)
- Natural Science Foundation of Tianjin Municipality (20JCZDJC00180, 19JCZDJC31500)
- National Laboratory of Pattern Recognition (202000002)
This publication has 38 references indexed in Scilit:
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Fully convolutional networks for semantic segmentation2015
- Deep learningNature, 2015
- Ground-Based Cloud Detection Using Automatic Graph CutIEEE Geoscience and Remote Sensing Letters, 2015
- Neural Networks and Support Vector Machine Algorithms for Automatic Cloud Classification of Whole-Sky Ground-Based ImagesIEEE Geoscience and Remote Sensing Letters, 2014
- Salient local binary pattern for ground-based cloud classificationJournal of Meteorological Research, 2013
- Automatic cloud classification of whole sky imagesAtmospheric Measurement Techniques, 2010
- Retrieving Cloud Characteristics from Ground-Based Daytime Color All-Sky ImagesJournal of Atmospheric and Oceanic Technology, 2006
- Cloud-Base-Height Estimation from Paired Ground-Based Hemispherical ObservationsJournal of Applied Meteorology and Climatology, 2005
- The computation of cloud base height from paired whole-sky imaging camerasMachine Vision and Applications, 1997