RT-DETR-MCDAF: Multimodal Fusion of Visible Light and Near-Infrared Images for Citrus Surface Defect Detection in the Compound Domain
Open Access
- 17 March 2025
- journal article
- research article
- Published by MDPI AG in Agriculture
- Vol. 15 (6) , 630
Abstract
The accurate detection of citrus surface defects is essential for automated citrus sorting to enhance the commercialization of the citrus industry. However, previous studies have only focused on single-modal defect detection using visible light images (RGB) or near-infrared light images (NIR), without considering the feature fusion between these two modalities. This study proposed an RGB-NIR multimodal fusion method to extract and integrate key features from both modalities to enhance defect detection performance. First, an RGB-NIR multimodal dataset containing four types of citrus surface defects (cankers, pests, melanoses, and cracks) was constructed. Second, a Multimodal Compound Domain Attention Fusion (MCDAF) module was developed for multimodal channel fusion. Finally, MCDAF was integrated into the feature extraction network of Real-Time DEtection TRansformer (RT-DETR). The experimental results demonstrated that RT-DETR-MCDAF achieved Precision, Recall, mAP@0.5, and mAP@0.5:0.95 values of 0.914, 0.919, 0.90, and 0.937, respectively, with an average detection performance of 0.598. Compared with the model RT-DETR-RGB&NIR, which used simple channel concatenation fusion, RT-DETR-MCDAF improved the performance by 1.3%, 1.7%, 1%, 1.5%, and 1.7%, respectively. Overall, the proposed model outperformed traditional channel fusion methods and state-of-the-art single-modal models, providing innovative insights for commercial citrus sorting.Keywords
Funding Information
- Distinguished Young Scholars Fund of the Hunan Provincial Natural Science Foundation (2023JJ10099)
This publication has 29 references indexed in Scilit:
- An improved YOLOv7 network using RGB-D multi-modal feature fusion for tea shoots detectionComputers and Electronics in Agriculture, 2023
- YOLACTFusion: An instance segmentation method for RGB-NIR multimodal image fusion based on an attention mechanismComputers and Electronics in Agriculture, 2023
- Where's your head at? Detecting the orientation and position of pigs with rotated bounding boxesComputers and Electronics in Agriculture, 2023
- A method of citrus epidermis defects detection based on an improved YOLOv5Biosystems Engineering, 2023
- Real-time defects detection for apple sorting using NIR cameras with pruning-based YOLOV4 networkComputers and Electronics in Agriculture, 2022
- Computer vision based detection of external defects on tomatoes using deep learningBiosystems Engineering, 2020
- Automatic Detection and Grading of Multiple Fruits by Machine LearningFood Analytical Methods, 2019
- Detection of Early Rottenness on Apples by Using Hyperspectral Imaging Combined with Spectral Analysis and Image ProcessingFood Analytical Methods, 2015
- Computer-based detection and classification of flaws in citrus fruitsNeural Computing and Applications, 2010
- Automatic detection of skin defects in citrus fruits using a multivariate image analysis approachComputers and Electronics in Agriculture, 2010