Intrinsic Explainability of Multimodal Learning for Crop Yield Prediction
Authors:
Preprint
Abstract
Multimodal learning enables various machine learning tasks to benefit from diverse data sources, effectively mimicking the interplay of different factors in real-world applications, particularly in agriculture. While the heterogeneous nature of involved data modalities may necessitate the design of complex architectures, the model interpretability is often overlooked. In this study, we leverage the intrinsic explainability of Transformer-based models to explain multimodal learning networks, focusing on the task of crop yield prediction at the subfield level. The large datasets used cover various crops, regions, and years, and include four different input modalities: multispectral satellite and weather time series, terrain elevation maps and soil properties. Based on the self-attention mechanism, we estimate feature attributions using two methods, namely the Attention Rollout (AR) and Generic Attention (GA), and evaluate their performance against Shapley-based model-agnostic estimations, Shapley Value Sampling (SVS). Additionally, we propose the Weighted Modality Activation (WMA) method to assess modality attributions and compare it with SVS attributions. Our findings indicate that Transformer-based models outperform other architectures, specifically convolutional and recurrent networks, achieving R2 scores that are higher by 0.10 and 0.04 at the subfield and field levels, respectively. AR is shown to provide more robust and reliable temporal attributions, as confirmed through qualitative and quantitative evaluation, compared to GA and SVS values. Information about crop phenology stages was leveraged to interpret the explanation results in the light of established agronomic knowledge. Furthermore, modality attributions revealed varying patterns across the two methods compared.[...]Keywords
All Related Versions
This publication has 47 references indexed in Scilit:
- Machine learning technology for early prediction of grain yield at the field scale: A systematic reviewComputers and Electronics in Agriculture, 2023
- Interpretability of deep learning models for crop yield forecastingComputers and Electronics in Agriculture, 2023
- Extreme Gradient Boosting for yield estimation compared with Deep Learning approachesComputers and Electronics in Agriculture, 2022
- Combined analysis of satellite and ground data for winter wheat yield forecastingSmart Agricultural Technology, 2022
- DeepYield: A combined convolutional neural network with long short-term memory for crop yield forecastingExpert Systems with Applications, 2021
- Integrating Multi-Source Data for Rice Yield Prediction across China using Machine Learning and Deep Learning ApproachesAgricultural and Forest Meteorology, 2021
- The ERA5 global reanalysisQuarterly Journal of the Royal Meteorological Society, 2020
- Crop yield prediction with deep convolutional neural networksComputers and Electronics in Agriculture, 2019
- Integrating satellite and climate data to predict wheat yield in Australia using machine learning approachesAgricultural and Forest Meteorology, 2019
- Deep convolutional neural networks for rice grain yield estimation at the ripening stage using UAV-based remotely sensed imagesField Crops Research, 2019