Revisiting Modality Imbalance In Multimodal Pedestrian Detection
Authors:
Preprint
Abstract
Multimodal learning, particularly for pedestrian detection, has recently received emphasis due to its capability to function equally well in several critical autonomous driving scenarios such as low-light, night-time, and adverse weather conditions. However, in most cases, the training distribution largely emphasizes the contribution of one specific input that makes the network biased towards one modality. Hence, the generalization of such models becomes a significant problem where the non-dominant input modality during training could be contributing more to the course of inference. Here, we introduce a novel training setup with regularizer in the multimodal architecture to resolve the problem of this disparity between the modalities. Specifically, our regularizer term helps to make the feature fusion method more robust by considering both the feature extractors equivalently important during the training to extract the multimodal distribution which is referred to as removing the imbalance problem. Furthermore, our decoupling concept of output stream helps the detection task by sharing the spatial sensitive information mutually. Extensive experiments of the proposed method on KAIST and UTokyo datasets shows improvement of the respective state-of-the-art performance.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions2021
- Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems2020
- TiledSoilingNet: Tile-level Soiling Detection on Automotive Surround-view Cameras Using Coverage Metric2020
- Attention Based Multi-Layer Fusion of Multispectral Images for Pedestrian DetectionIEEE Access, 2020
- Weakly Aligned Cross-Modal Learning for Multispectral Pedestrian Detection2019
- RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving2019
- Repulsion Loss: Detecting Pedestrians in a Crowd2018
- Aggregated Residual Transformations for Deep Neural Networks2017
- Deep Residual Learning for Image Recognition2016
- Multispectral pedestrian detection: Benchmark dataset and baseline2015