Multi-View And Multi-Modal Event Detection Utilizing Transformer-Based Multi-Sensor Fusion
Authors:
- 23 May 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4638-4642
Abstract
We tackle a challenging task: multi-view and multi-modal event detection that detects events in a wide-range real environment by utilizing data from distributed cameras and microphones and their weak labels. In this task, distributed sensors are utilized complementarily to capture events that are difficult to capture with a single sensor, such as a series of actions of people moving in an intricate room, or communication between people located far apart in a room. For sensors to cooperate effectively in such a situation, the system should be able to exchange information among sensors and combines information that is useful for identifying events in a complementary manner. For such a mechanism, we propose a Transformer-based multi-sensor fusion (MultiTrans) which combines multi-sensor data on the basis of the relationships between features of different viewpoints and modalities. In the experiments using a dataset 1 newly collected for this task, our proposed method using MultiTrans improved the event detection performance and outperformed comparatives.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Multiple instance learning: A survey of problem characteristics and applicationsPattern Recognition, 2018
- Recognition of emergency situations using audio–visual perception sensor network for ambient assistive livingJournal of Ambient Intelligence and Humanized Computing, 2017
- Multi-view 3D Object Detection Network for Autonomous Driving2017
- Spatial Cepstrum as a Spatial Feature Using a Distributed Microphone Array for Acoustic Scene AnalysisIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2017
- CNN architectures for large-scale audio classification2017
- TUT database for acoustic scene classification and sound event detection2016
- NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis2016
- Deep Residual Learning for Image Recognition2016
- Multi-view Convolutional Neural Networks for 3D Shape Recognition2015
- Cross-View Action Modeling, Learning, and Recognition2014