Cross-Modal Data Fusion via Vision-Language Model for Crop Disease Recognition
Open Access
Abstract
Crop diseases pose a significant threat to agricultural productivity and global food security. Timely and accurate disease identification is crucial for improving crop yield and quality. While most existing deep learning-based methods focus primarily on image datasets for disease recognition, they often overlook the complementary role of textual features in enhancing visual understanding. To address this problem, we proposed a cross-modal data fusion via a vision-language model for crop disease recognition. Our approach leverages the Zhipu.ai multi-model to generate comprehensive textual descriptions of crop leaf diseases, including global description, local lesion description, and color-texture description. These descriptions are encoded into feature vectors, while an image encoder extracts image features. A cross-attention mechanism then iteratively fuses multimodal features across multiple layers, and a classification prediction module generates classification probabilities. Extensive experiments on the Soybean Disease, AI Challenge 2018, and PlantVillage datasets demonstrate that our method outperforms state-of-the-art image-only approaches with higher accuracy and fewer parameters. Specifically, with only 1.14M model parameters, our model achieves a 98.74%, 87.64% and 99.08% recognition accuracy on the three datasets, respectively. The results highlight the effectiveness of cross-modal learning in leveraging both visual and textual cues for precise and efficient disease recognition, offering a scalable solution for crop disease recognition.Keywords
Funding Information
- National Natural Science Foundation of China (62301287, 23KJB510025)
- Natural Science Foundation of the Jiangsu Higher Education Institutions of China (62301287, 23KJB510025)
This publication has 30 references indexed in Scilit:
- Transforming agriculture with Machine Learning, Deep Learning, and IoT: perspectives from Ethiopia—challenges and opportunitiesDiscover Agriculture, 2024
- Deep learning implementation of image segmentation in agricultural applications: a comprehensive reviewArtificial Intelligence Review, 2024
- VGG-ICNN: A Lightweight CNN model for crop disease identificationMultimedia Tools and Applications, 2022
- Research on plant disease identification based on CNNCognitive Robotics, 2022
- Crop disease identification and interpretation method based on multimodal deep learningComputers and Electronics in Agriculture, 2021
- L-CSMS: novel lightweight network for plant disease severity recognitionJournal of Plant Diseases and Protection, 2021
- Research on deep learning in apple leaf disease recognitionComputers and Electronics in Agriculture, 2020
- Attention embedded residual CNN for disease detection in tomato leavesApplied Soft Computing, 2020
- Plant disease identification from individual lesions and spots using deep learningBiosystems Engineering, 2019
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design2018