Fusion of Satellite Images and Weather Data With Transformer Networks for Downy Mildew Disease Detection

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Crop diseases significantly affect the quantity and quality of agricultural production. In a context where the goal of precision agriculture is to minimize or even avoid the use of pesticides, weather and remote sensing data with deep learning can play a pivotal role in detecting crop diseases, allowing localized treatment of crops. However, combining heterogeneous data such as weather and images remains a hot topic and challenging task. Recent developments in transformer architectures have shown the possibility of fusion of data from different domains, such as text-image. The current trend is to custom only one transformer to create a multimodal fusion model. Conversely, we propose a new approach to realize data fusion using three transformers. In this paper, we first solved the missing satellite images problem, by interpolating them with a ConvLSTM model. Then, proposed a multimodal fusion architecture that jointly learns to process visual and weather information. The architecture is built from three main components, a Vision Transformer and two transformer-encoders, allowing to fuse both image and weather modalities. The results of the proposed method are promising achieving an overall accuracy of 97%.
Funding Information
  • European Consortium ERA-NET ICT-AGRI-FOOD
  • French National Research Agency (ANR-21-ICAF-0002-01)
  • European Project MERIAVINO
  • ICT-AGRIFOOD ERA-NET

This publication has 38 references indexed in Scilit: