Alexis Carlier

Computer vision is increasingly used in farmers’ fields and agricultural experiments to quantify important traits. Imaging setups with a sub-millimetre ground sampling distance enable the detection and tracking of plant features, including size, shape, and colour. Although today’s AI-driven foundation models segment almost any object in an image, they still fail for complex plant canopies. To improve model performance, the global wheat dataset consortium assembled a diverse set of images from experiments around the globe. After the head detection dataset (GWHD), the new dataset targets a full semantic segmentation (GWFSS) of wheat organs (leaves, stems and spikes) covering all developmental stages. Images were collected by 11 institutions using a wide range of imaging setups. Two datasets are provided: i) a set of 1096 diverse images in which all organs were labelled at the pixel level, and (ii) a dataset of 52,078 images without annotations available for additional training. The labelled set was used to train segmentation models based on DeepLabV3Plus and Segformer. Our Segformer model performed slightly better than DeepLabV3Plus with a mIOU for leaves and spikes of ca. 90%. However, the precision for stems with 54% was rather lower. The major advantages over published models are: i) the exclusion of weeds from the wheat canopy, ii) the detection of all wheat features including necrotic and senescent tissues and its separation from crop residues. This facilitates further development in classifying healthy vs. unhealthy tissue to address the increasing need for accurate quantification of senescence and diseases in wheat canopies.

The utilization of high-throughput in-field phenotyping systems presents new opportunities for evaluating crop stress. However, existing studies have primarily focused on individual stresses, overlooking the fact that crops in field conditions frequently encounter multiple stresses, which can display similar symptoms or interfere with the detection of other stress factors. Therefore, this study aimed to investigate the impact of wheat yellow rust on reflectance measurements and nitrogen status assessment. A multi-sensor mobile platform was utilized to capture RGB and multispectral images throughout a 2-year fertilization-fungicide trial. To identify disease-induced damage, the SegVeg approach, which combines a U-NET architecture and a pixel-wise classifier, was applied to RGB images, generating a mask capable of distinguishing between healthy and damaged areas of the leaves. The observed proportion of damage in the images demonstrated similar effectiveness to visual scoring methods in explaining grain yield. Furthermore, the study discovered that the disease not only affected reflectance through leaf damage but also influenced the reflectance of healthy areas by disrupting the overall nitrogen status of the plants. This emphasizes the importance of incorporating disease impact into reflectance-based decision support tools to account for its effects on spectral data. This effect was successfully mitigated by employing the NDRE vegetation index calculated exclusively from the healthy portions of the leaves or by incorporating the proportion of damage into the model. However, these findings also highlight the necessity for further research specifically addressing the challenges presented by multiple stresses in crop phenotyping.

The reflectance of wheat crops provides information on their architecture or physiology. However, the methods currently used for close-range reflectance computation do not allow for the separation of the wheat canopy organs: the leaves and the ears. This study details a method to achieve high-throughput measurements of wheat reflectance at the organ scale. A nadir multispectral camera array and an incident light spectrometer were used to compute bi-directional reflectance factor (BRF) maps. Image thresholding and deep learning ear detection allowed for the segmentation of the ears and the leaves in the maps. The results showed that the BRF measured on reference targets was constant throughout the day but varied with the acquisition date. The wheat organ BRF was constant throughout the day in very cloudy conditions and with high sun altitudes but showed gradual variations in the morning under sunny or partially cloudy sky. As a consequence, measurements should be performed close to solar noon and the reference panel should be captured at the beginning and end of each field trip to correct the BRF. The method, with such precautions, was tested all throughout the wheat growing season on two varieties and various canopy architectures generated by a fertilization gradient. The method yielded consistent reflectance dynamics in all scenarios.

The automatic segmentation of ears in wheat canopy images is an important step to measure ear density or extract relevant plant traits separately for the different organs. Recent deep learning algorithms appear as promising tools to accurately detect ears in a wide diversity of conditions. However, they remain complicated to implement and necessitate a huge training database. This paper is aimed at proposing an easy and quick to train and robust alternative to segment wheat ears from heading to maturity growth stage. The tested method was based on superpixel classification exploiting features from RGB and multispectral cameras. Three classifiers were trained with wheat images acquired from heading to maturity on two cultivars at different levels of fertilizer. The best classifier, the support vector machine (SVM), yielded satisfactory segmentation and reached 94 accuracy. However, the segmentation at the pixel level could not be assessed only by the superpixel classification accuracy. For this reason, a second assessment method was proposed to consider the entire process. A simple graphical tool was developed to annotate pixels. The strategy was to annotate a few pixels per image to be able to quickly annotate the entire image set, and thus account for very diverse conditions. Results showed a lesser segmentation score (F1-score) for the heading and flowering stages and for the zero nitrogen input object. The methodology appeared appropriate for further work on the growth dynamics of the different wheat organs and in the frame of other segmentation challenges.

Large pre-trained language models have shown promise for few-shot learning, completing text-based tasks given only a few task-specific examples. Will models soon solve classification tasks that have so far been reserved for human research assistants? Existing benchmarks are not designed to measure progress in applied settings, and so don't directly answer this question. The RAFT benchmark (Real-world Annotated Few-shot Tasks) focuses on naturally occurring tasks and uses an evaluation setup that mirrors deployment. Baseline evaluations on RAFT reveal areas current techniques struggle with: reasoning over long texts and tasks with many classes. Human baselines show that some classification tasks are difficult for non-expert humans, reflecting that real-world value sometimes depends on domain expertise. Yet even non-expert human baseline F1 scores exceed GPT-3 by an average of 0.11. The RAFT datasets and leaderboard will track which model improvements translate into real-world benefits at https://raft.elicit.org .

Multimodal images fusion has the potential to enrich the information gathered by multi-sensor plant phenotyping platforms. Fusion of images from multiple sources is, however, hampered by the technical lock of image registration. The aim of this paper is to provide a solution to the registration and fusion of multimodal wheat images in field conditions and at close range. Eight registration methods were tested on nadir wheat images acquired by a pair of red, green and blue (RGB) cameras, a thermal camera and a multispectral camera array. The most accurate method, relying on a local transformation, aligned the images with an average error of 2 mm but was not reliable for thermal images. More generally, the suggested registration method and the preprocesses necessary before fusion (plant mask erosion, pixel intensity averaging) would depend on the application. As a consequence, the main output of this study was to identify four registration-fusion strategies: (i) the REAL-TIME strategy solely based on the cameras’ positions, (ii) the FAST strategy suitable for all types of images tested, (iii) and (iv) the ACCURATE and HIGHLY ACCURATE strategies handling local distortion but unable to deal with images of very different natures. These suggestions are, however, limited to the methods compared in this study. Further research should investigate how recent cutting-edge registration methods would perform on the specific case of wheat canopy.

The Global Wheat Head Detection (GWHD) dataset was created in 2020 and has assembled 193,634 labelled wheat heads from 4700 RGB images acquired from various acquisition platforms and 7 countries/institutions. With an associated competition hosted in Kaggle, GWHD_2020 has successfully attracted attention from both the computer vision and agricultural science communities. From this first experience, a few avenues for improvements have been identified regarding data size, head diversity, and label reliability. To address these issues, the 2020 dataset has been reexamined, relabeled, and complemented by adding 1722 images from 5 additional countries, allowing for 81,553 additional wheat heads. We now release in 2021 a new version of the Global Wheat Head Detection dataset, which is bigger, more diverse, and less noisy than the GWHD_2020 version.