Harnessing Multi-Modal Synergy: A Systematic Framework for Disaster Loss Consistency Analysis and Emergency Response
Open Access
Abstract
When a disaster occurs, a large number of social media posts on platforms like Weibo attract public attention with their combination of text and images. However, the consistency between textual descriptions and visual representations varies significantly. Consistent multi-modal data are crucial for helping the public understand the disaster situation and support rescue efforts. This study aims to develop a systematic framework for assessing the consistency of multi-modal disaster-related data on social media. This study explored how the congruence between text and image content affects public engagement and informs strategies for efficient emergency responses. Firstly, the Clip (Contrastive Language-Image Pre-Training) model was used to mine the disaster correlation, loss category, and severity of the images and text. Then, the consistency of image–text pairs was qualitatively analyzed and quantitatively calculated. Finally, the influence of graphic consistency on social concern was discussed. The experimental findings reveal that the consistency of text and image data significantly influences the degree of public concern. When the consistency increases by 1%, the social attention index will increase by about 0.8%. This shows that consistency is a key factor for attracting public attention and promoting the dissemination of information related to important disasters. The proposed framework offers a robust, systematic approach to analyzing disaster loss information consistency. It allows for the efficient extraction of high-consistency data from vast social media data sets, providing governments and emergency response agencies with timely, accurate insights into disaster situations.Keywords
Funding Information
- National Natural Science Foundation of China (72071010)
This publication has 37 references indexed in Scilit:
- A Model for Classifying Emergency Events Based on Social Media Multimodal Data2023
- Image-based preliminary emergency assessment of damaged buildings after earthquake: Taiwan case studiesEngineering Applications of Artificial Intelligence, 2023
- Real-time social media sentiment analysis for rapid impact assessment of floodsComputers & Geosciences, 2023
- An ALBERT-based TextCNN-Hatt hybrid model enhanced with topic knowledge for sentiment analysis of sudden-onset disastersEngineering Applications of Artificial Intelligence, 2023
- An integrated convolutional neural network and sorting algorithm for image classification for efficient flood disaster managementDecision Analytics Journal, 2023
- Exploring the impact of sentiment on multi-dimensional information dissemination using COVID-19 data in ChinaComputers in Human Behavior, 2023
- Multi-label disaster text classification via supervised contrastive learning for social media dataComputers and Electrical Engineering, 2022
- Multi-modal classification of Twitter data during disasters for humanitarian responseJournal of Ambient Intelligence and Humanized Computing, 2021
- A deep multi-modal neural network for informative Twitter content classification during emergenciesAnnals of Operations Research, 2020
- Citizens as sensors: the world of volunteered geographyGeoJournal, 2007