LLM-Enhanced multimodal detection of fake news
Open Access
- 24 October 2024
- journal article
- research article
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 19 (10) , e0312240
Abstract
Fake news detection is growing in importance as a key topic in the information age. However, most current methods rely on pre-trained small language models (SLMs), which face significant limitations in processing news content that requires specialized knowledge, thereby constraining the efficiency of fake news detection. To address these limitations, we propose the FND-LLM Framework, which effectively combines SLMs and LLMs to enhance their complementary strengths and explore the capabilities of LLMs in multimodal fake news detection. The FND-LLM framework integrates the textual feature branch, the visual semantic branch, the visual tampering branch, the co-attention network, the cross-modal feature branch and the large language model branch. The textual feature branch and visual semantic branch are responsible for extracting the textual and visual information of the news content, respectively, while the co-attention network is used to refine the interrelationship between the textual and visual information. The visual tampering branch is responsible for extracting news image tampering features. The cross-modal feature branch enhances inter-modal complementarity through the CLIP model, while the large language model branch utilizes the inference capability of LLMs to provide auxiliary explanation for the detection process. Our experimental results indicate that the FND-LLM framework outperforms existing models, achieving improvements of 0.7%, 6.8% and 1.3% improvements in overall accuracy on Weibo, Gossipcop, and Politifact, respectively.Keywords
Funding Information
- Zhejiang Provincial Natural Science Foundation of China (LY24F020004)
- National Social Science Foundation of China (19BXW003)
- Humanities and Social Sciences Research Project of the Ministry of Education of China (19YJC740090)
- Zhejiang Gongshang University ‘`Digital+" Disciplinary Construction Management Project (SZJ2022B016)
- Zhejiang University Students Science and Technology Innovation Activity Plan and New Talent Plan (2023R408076)
This publication has 44 references indexed in Scilit:
- A review of fake news detection approaches: A critical analysis of relevant studies and highlighting key challenges associated with the dataset, feature representation, and data fusionHeliyon, 2023
- A rumor heat prediction model based on rumor and anti-rumor multiple messages and knowledge representationInformation Processing & Management, 2023
- Multimodal fake news detection via progressive fusion networksInformation Processing & Management, 2022
- BCMF: A bidirectional cross-modal fusion model for fake news detectionInformation Processing & Management, 2022
- Detecting fake news by exploring the consistency of multimodal dataInformation Processing & Management, 2021
- Exploring the Role of Visual Content in Fake News Detection2020
- FakeNewsNet: A Data Repository with News Content, Social Context, and Spatiotemporal Information for Studying Fake News on Social MediaBig Data, 2020
- $\mathsf {SAFE}$: Similarity-Aware Multi-modal Fake News Detection2020
- Going Deeper in Spiking Neural Networks: VGG and Residual ArchitecturesFrontiers in Neuroscience, 2019
- Call Attention to Rumors: Deep Attention Based Recurrent Neural Networks for Early Rumor Detection2018