Cloud-Type Classification Using Vision-Language Model With Cloud Multimodal Adapter
- 8 May 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 64 (01962892) , 5621413
Abstract
The existing methods for cloud-type classification primarily rely on cloud observation data containing low-level semantic information, e.g., cloud images and millimeter-wave cloud radar (MMCR) observations, which limits the ability to bridge the gap between original cloud data and human cognitive understanding. In this paper, we apply Large Language Model (LLM) to generate cloud description texts which include high-level semantic information, and we integrate cloud images, MMCR observations and cloud description texts for cloud-type classification. To effectively fuse the information across modalities, we propose a novel adapter named Cloud Multi-Modal Adapter (CMMA) for Vision Language Model (VLM) fine-tuning. The proposed CMMA could adaptively fuse the image features with both radar and text features extracted from corresponding adapters, which enhances the fusion across different modalities and improves the model’s ability to learn from multi-modal data. Meanwhile, we propose the Tri-Modalities Contrastive Loss (TCL) to enhance cross-modal alignment while preserving intra-modal consistency. Extensive experiments demonstrate the effectiveness of our approach.Keywords
Funding Information
- National Natural Science Foundation of China (62171321)
- National Natural Science Foundation of China (62471335)
This publication has 62 references indexed in Scilit:
- Learning Knowledge Enhanced Text-image Feature Selection Network for Chest X-ray Image ClassificationMachine Intelligence Research, 2025
- Multimodal Pretrained Knowledge for Real-world Object NavigationMachine Intelligence Research, 2025
- Assessing and Understanding Creativity in Large Language ModelsMachine Intelligence Research, 2025
- CloudSense: A model for cloud type identification using machine learning from radar dataApplied Computing and Geosciences, 2024
- Multimodal Fusion of Brain Imaging Data: Methods and ApplicationsMachine Intelligence Research, 2024
- CLIP-Adapter: Better Vision-Language Models with Feature AdaptersInternational Journal of Computer Vision, 2023
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Automatic Identification of Clear-Air Echoes Based on Millimeter-wave Cloud Radar MeasurementsAdvances in Atmospheric Sciences, 2020
- Cloud Classification and Distribution of Cloud Types in Beijing Using Ka-Band Radar DataAdvances in Atmospheric Sciences, 2019
- Meteorological research applications of MM-wave radarMeteorology and Atmospheric Physics, 1996