Abstract
Recently multimodal learning has become a popular research topic because variety of types of sensors can be used to describe the events, for example, images and texts in the social network for disaster detection and response. It is useful because of different sources of information that data will cause heterogeneous problems. The kernel function can measure the similarity of data, and the fusion of different kernels can handle this problem well. This paper proposes a method based on multimodal deep kernel learning to detect specific categories of content posted on Twitter during a disaster for humanitarian response. Specifically, the convolutional neural network is used to extract image and text features, and obtain multiple kernel functions in each mode. Then a deep neural network is used to fuse multiple basic kernel functions corresponding to the two modalities. Finally, we use Support Vector Machine (SVM) for classification. Applying this method to a real disaster data set, and compared with the baseline, the results prove the effectiveness of our model.
Funding Information
  • National Natural Science Foundation of China (61806180)

This publication has 13 references indexed in Scilit: