Adaptive Knowledge Distillation With Attention-Based Multi-Modal Fusion for Robust Dim Object Detection
- 24 December 2024
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 2083-2096
Abstract
Automated object detection in aerial images is crucial in both civil and military applications. Existing computer vision-based object detection methods are not robust enough to precisely detect dim objects in aerial images due to the cluttered backgrounds, various observing angles, small object scales, and severe occlusions. Recently, electroencephalography (EEG)-based object detection methods have received increasing attention owing to the advanced cognitive capabilities of human vision. However, how to combine the human intelligence with computer intelligence to achieve robust dim object detection is still an open question. In this paper, we propose a novel approach to efficiently fuse and exploit the properties of multi-modal data for dim object detection. Specifically, we first design a brain-computer interface (BCI) paradigm called eye-tracking-based slow serial visual presentation (ESSVP) to simultaneously collect the paired EEG and image data when subjects search for the dim objects in aerial images. Then, we develop an attention-based multi-modal fusion network to selectively aggregate the learned features of EEG and image modalities. Furthermore, we propose an adaptive multi-teacher knowledge distillation method to efficiently train the multi-modal dim object detector for better performance. To evaluate the effectiveness of our method, we conduct extensive experiments on the collected dataset in subject-dependent and subject-independent tasks. The experimental results demonstrate that the proposed dim object detection method exhibits superior effectiveness and robustness compared to the baselines and the state-of-the-art methods.Keywords
Funding Information
- National Natural Science Foundation of China (62403240)
- Natural Science Foundation of Jiangsu Province (BK20241396)
This publication has 49 references indexed in Scilit:
- Cross-modal guiding and reweighting network for multi-modal RSVP-based target detectionNeural Networks, 2023
- Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognitionPattern Recognition, 2022
- 4D attention-based neural network for EEG emotion recognitionCognitive Neurodynamics, 2022
- The Necessity of Leave One Subject Out (LOSO) Cross Validation for EEG Disease Diagnosis2021
- Knowledge Distillation: A SurveyInternational Journal of Computer Vision, 2021
- Event-related components are structurally represented by intrinsic event-related potentialsScientific Reports, 2021
- Vehicle tracking by detection in UAV aerial videoScience China Information Sciences, 2019
- A review of rapid serial visual presentation-based brain–computer interfacesJournal of Neural Engineering, 2018
- PsychoPy—Psychophysics software in PythonJournal of Neuroscience Methods, 2007
- EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysisJournal of Neuroscience Methods, 2004