Multimodal Object Detection via Probabilistic Ensembling
Preprint
Abstract
Object detection with multimodal inputs can improve many safety-critical systems such as autonomous vehicles (AVs). Motivated by AVs that operate in both day and night, we study multimodal object detection with RGB and thermal cameras, since the latter provides much stronger object signatures under poor illumination. We explore strategies for fusing information from different modalities. Our key contribution is a probabilistic ensembling technique, ProbEn, a simple non-learned method that fuses together detections from multi-modalities. We derive ProbEn from Bayes' rule and first principles that assume conditional independence across modalities. Through probabilistic marginalization, ProbEn elegantly handles missing modalities when detectors do not fire on the same object. Importantly, ProbEn also notably improves multimodal detection even when the conditional independence assumption does not hold, e.g., fusing outputs from other fusion methods (both off-the-shelf and trained in-house). We validate ProbEn on two benchmarks containing both aligned (KAIST) and unaligned (FLIR) multimodal images, showing that ProbEn outperforms prior work by more than 13% in relative performance!Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- Weighted boxes fusion: Ensembling boxes from different object detection modelsImage and Vision Computing, 2021
- Task-Conditioned Domain Adaptation for Pedestrian Detection in Thermal Imagery2020
- Fusion of multispectral data through illumination-aware deep neural networks for pedestrian detectionInformation Fusion, 2019
- Cross-modality interactive attention network for multispectral pedestrian detectionInformation Fusion, 2019
- Multi-style Generative Network for Real-Time Transfer2019
- Illumination-aware faster R-CNN for robust multispectral pedestrian detectionPattern Recognition, 2019
- The Pascal Visual Object Classes Challenge: A RetrospectiveInternational Journal of Computer Vision, 2014
- Microsoft COCO: Common Objects in Context2014
- Histograms of Oriented Gradients for Human Detection2005
- An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and VariantsMachine Learning, 1999