Cross-modal Contrastive Learning for Multimodal Fake News Detection
Open Access
- 26 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 5696-5704
Abstract
Automatic detection of multimodal fake news has gained a widespread attention recently. Many existing approaches seek to fuse unimodal features to produce multimodal news representations. However, the potential of powerful cross-modal contrastive learning methods for fake news detection has not been well exploited. Besides, how to aggregate features from different modalities to boost the performance of the decision-making process is still an open question. To address that, we propose COOLANT, a cross-modal contrastive learning framework for multimodal fake news detection, aiming to achieve more accurate image-text alignment. To further capture the fine-grained alignment between vision and language, we leverage an auxiliary task to soften the loss term of negative samples during the contrast process. A cross-modal fusion module is developed to learn the cross-modality correlations. An attention mechanism with an attention guidance module is implemented to help effectively and interpretably aggregate the aligned unimodal representations and the cross-modality correlations. Finally, we evaluate the COOLANT and conduct a comparative study on two widely used datasets, Twitter and Weibo. The experimental results demonstrate that our COOLANT outperforms previous approaches by a large margin and achieves new state-of-the-art results on the two datasets.Keywords
All Related Versions
Funding Information
- Strategic Priority Research Program of Chinese Academy of Sciences (No. XDC02040400)
- National Key Research and Development of China (No. 2021YFB3100600)
This publication has 17 references indexed in Scilit:
- Multi-modal Knowledge-aware Event Memory Network for Social Media Rumor Detection2019
- Rumor Detection with Hierarchical Social Attention Network2018
- EANN2018
- Squeeze-and-Excitation Networks2018
- Detection and Resolution of Rumours in Social MediaACM Computing Surveys, 2018
- Detection and visualization of misleading content on TwitterInternational Journal of Multimedia Information Retrieval, 2017
- Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs2017
- Deep Residual Learning for Image Recognition2016
- Real-time Rumor Debunking on Twitter2015
- Automatic deception detection: Methods for finding fake newsProceedings of the Association for Information Science and Technology, 2015