Hierarchical Fusion Transformer for Multimodal Ground-Based Cloud Type Classification
Open Access
- 26 September 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
- Vol. 18 (19391404) , 25192-25203
Abstract
Existing methods for multimodal ground-based cloud type classification are dominated by Convolutional Neural Networks (CNNs), and it fails to capture long-range dependencies. In this paper, we propose a novel Transformer-based architecture named Hierarchical Fusion Transformer (HFT) for multimodal ground-based cloud type classification, which leverages the advantages of self-attention and cross-attention to learn long-range dependencies and effectively fuse cloud images and meteorological element information. Specifically, we propose Visual and Meteorological Joint-Transformer (VM Joint-Trans) to capture global context across modalities, and present Visual and Meteorological Cross-Transformer (VM Cross-Trans) to align different modalities and reduce their inconsistencies. We design a hierarchical architecture to perform comprehensive fusion using VM Joint-Trans and VM Cross-Trans. Meanwhile, we propose the novel Multimodal Contrastive Learning (MCL) which not only constrains the tokens of cloud images and meteorological element information in the same layer, but also the tokens from the same modality in different layers, thereby improving the discriminative ability of model and reducing the modality gap. Furthermore, we release the Large-scale Multimodal Ground-based Cloud Database (LMGCD), containing 10,000 multimodal samples with seven categories. To the best of our knowledge, it is the largest database for multimodal ground-based cloud type classification. Experimental results validate the effectiveness of the proposed HFT for multimodal ground-based cloud type classification.Keywords
Funding Information
- National Natural Science Foundation of China (62471335)
- National Natural Science Foundation of China (62171321)
- National Natural Science Foundation of China (62206095)
- S&T Program of Hebei (SZX2020034)
This publication has 55 references indexed in Scilit:
- Global distributions of multi-layer and multi-phase clouds and their cloud radiative effectsJournal of Quantitative Spectroscopy and Radiative Transfer, 2024
- In the Eye of Transformer: Global–Local Correlation for Egocentric Gaze Estimation and BeyondInternational Journal of Computer Vision, 2023
- A transformer-based representation-learning model with unified processing of multimodal input for clinical diagnosticsNature Biomedical Engineering, 2023
- Urban effects on local cloud patternsProceedings of the National Academy of Sciences, 2023
- Cloud and cloud shadow detection for optical satellite imagery: Features, algorithms, validation, and prospectsISPRS Journal of Photogrammetry and Remote Sensing, 2022
- Contrasting impacts of forests on cloud cover based on satellite observationsNature Communications, 2022
- Clouds, radiation, and atmospheric circulation in the present‐day climate and under climate changeWires Climate Change, 2020
- Ground‐Based Cloud Classification Using Task‐Based Graph Convolutional NetworkGeophysical Research Letters, 2020
- A local binary pattern classification approach for cloud types derived from all-sky imagersInternational Journal of Remote Sensing, 2018
- CloudNet: Ground‐Based Cloud Classification With Deep Convolutional Neural NetworkGeophysical Research Letters, 2018