Abstract
The existing methods for cloud-type classification primarily rely on cloud observation data containing low-level semantic information, e.g., cloud images and millimeter-wave cloud radar (MMCR) observations, which limits the ability to bridge the gap between original cloud data and human cognitive understanding. In this paper, we apply Large Language Model (LLM) to generate cloud description texts which include high-level semantic information, and we integrate cloud images, MMCR observations and cloud description texts for cloud-type classification. To effectively fuse the information across modalities, we propose a novel adapter named Cloud Multi-Modal Adapter (CMMA) for Vision Language Model (VLM) fine-tuning. The proposed CMMA could adaptively fuse the image features with both radar and text features extracted from corresponding adapters, which enhances the fusion across different modalities and improves the model’s ability to learn from multi-modal data. Meanwhile, we propose the Tri-Modalities Contrastive Loss (TCL) to enhance cross-modal alignment while preserving intra-modal consistency. Extensive experiments demonstrate the effectiveness of our approach.
Funding Information
  • National Natural Science Foundation of China (62171321)
  • National Natural Science Foundation of China (62471335)

This publication has 62 references indexed in Scilit: