Multimodal Deep Kernel Learning: Application to Disaster Response in Social Media Network
- 24 September 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 560-564
Abstract
Recently multimodal learning has become a popular research topic because variety of types of sensors can be used to describe the events, for example, images and texts in the social network for disaster detection and response. It is useful because of different sources of information that data will cause heterogeneous problems. The kernel function can measure the similarity of data, and the fusion of different kernels can handle this problem well. This paper proposes a method based on multimodal deep kernel learning to detect specific categories of content posted on Twitter during a disaster for humanitarian response. Specifically, the convolutional neural network is used to extract image and text features, and obtain multiple kernel functions in each mode. Then a deep neural network is used to fuse multiple basic kernel functions corresponding to the two modalities. Finally, we use Support Vector Machine (SVM) for classification. Applying this method to a real disaster data set, and compared with the baseline, the results prove the effectiveness of our model.Keywords
Funding Information
- National Natural Science Foundation of China (61806180)
This publication has 13 references indexed in Scilit:
- Image–text sentiment analysis via deep multimodal attentive fusionKnowledge-Based Systems, 2019
- R-VQA2018
- End-to-End Convolutional Semantic Embeddings2018
- Learning Semantic Concepts and Order for Image and Sentence Matching2018
- Identification of Alzheimer's disease and mild cognitive impairment using multimodal sparse hierarchical extreme learning machineHuman Brain Mapping, 2018
- Processing Social Media Images by Combining Human and Machine Computing during CrisesInternational Journal of Human–Computer Interaction, 2018
- Nonlinear Deep Kernel Learning for Image AnnotationIEEE Transactions on Image Processing, 2017
- Deep kernel2016
- ModDrop: Adaptive Multi-Modal Gesture RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Processing Social Media Messages in Mass EmergencyACM Computing Surveys, 2015