Robust Multi-Modal Sensor Fusion: An Adversarial Approach
- 24 August 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Sensors Journal
- Vol. 21 (2) , 1885-1896
Abstract
In recent years, multi-modal fusion has attracted a lot of research interest, both in academia, and in industry. Multimodal fusion entails the combination of information from a set of different types of sensors. Exploiting complementary information from different sensors, we show that target detection and classification problems can greatly benefit from this fusion approach and result in a performance increase. To achieve this gain, the information fusion from various sensors is shown to require some principled strategy to ensure that additional information is constructively used, and has a positive impact on performance. We subsequently demonstrate the viability of the proposed fusion approach by weakening the strong dependence on the functionality of all sensors, hence introducing additional flexibility in our solution and lifting the severe limitation in unconstrained surveillance settings with potential environmental impact. Our proposed data driven approach to multimodal fusion, exploits selected optimal features from an estimated latent space of data across all modalities. This hidden space is learned via a generative network conditioned on individual sensor modalities. The hidden space, as an intrinsic structure, is then exploited in detecting damaged sensors, and in subsequently safeguarding the performance of the fused sensor system. Experimental results show that such an approach can achieve automatic system robustness against noisy/damaged sensors.Keywords
Funding Information
- U.S. Army Research Office (W911NF1910202)
This publication has 18 references indexed in Scilit:
- Learning with Side Information through Modality Hallucination2016
- Multimodal Task-Driven Dictionary Learning for Image ClassificationIEEE Transactions on Image Processing, 2015
- Subspace Interpolation via Dictionary Learning for Unsupervised Domain Adaptation2013
- Multisensor data fusion: A review of the state-of-the-artInformation Fusion, 2013
- Domain adaptation for object recognition: An unsupervised approach2011
- A Generalization of Bayesian Inference2008
- On the Detection of Footsteps Based on Acoustic and Seismic SensingAsilomar Conference on Signals, Systems & Computers, 2007
- Critiques on some combination rules for probability theory based on optimization techniques2007
- Convex optimization approach to identify fusion for multisensor target trackingIEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 2001
- An introduction to multisensor data fusionProceedings of the IEEE, 1997