Cross-modal retrieval aims to search relevant samples from one modality in response to a query of another modality. Mainstream methods typically rely on fully aligned datasets to construct contrastive losses for learning. However, for weakly aligned datasets, the lack of fine-grained alignment makes it challenging to learn effective representations for retrieval. In this work, we propose a method based on prompt learning. We design two types of prompts: Modality-specific Prompts and Shared Prompts. These prompts capture modality-specific information and shared semantic information, enabling multilevel interaction between modalities. Additionally, we use a description generation model to create matching text descriptions for images in each modality. These descriptions establish fine-grained alignment in the semantic space across different modalities. To ensure the model learns consistent and meaningful knowledge, We establish contrastive loss through multi-granularity alignment: (1) class-level, (2) instance-level within the same category, and (3) instance-level within different categories. The results show that our method achieves strong cross-modal retrieval performance even under weak alignment conditions.