Uni-Modal Event-Agnostic Knowledge Distillation for Multimodal Fake News Detection
- 10 October 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Knowledge and Data Engineering
- Vol. 36 (12) , 9490-9503
Abstract
With the rapid expansion of multimodal content in online social media, automatic detection of multimodal fake news has received much attention. Multimodal joint training commonly used in existing methods is expected to benefit from thoroughly leveraging cross-modal features, yet these methods still suffer from insufficient learning of uni-modal features. Due to the heterogeneity of multimodal networks, optimizing a single objective will inevitably make the models prone to rely on specific modality while leaving other modalities under-optimized. On the other hand, simply expecting each modality to play a significant role in identifying all the rumors is also not appropriate as the multimodal fake news often involves tampering in only one modality. Therefore, how to find the genuine tampering on the per-sample basis becomes the key point to unlock the full power of each modality in a good collaborative manner. To address these issues, we propose a Uni-modal Event-agnostic Knowledge Distillation framework framework (UEKD), which aims to transfer knowledge contained in the fine-grained prediction from uni-modal teachers to the multimodal student model through modality-specific distillation. Specifically, we find that the uni-modal teachers simply trained on the whole training set are easy to memorize the event-specific noise information to make a correct but biased prediction, failing to reflect the genuine degree of tampering in each modality. To tackle this problem, we propose to train and validate the teacher models on different domains in training dataset through a cross-validation manner, as the predictions from the out-of-domain teachers can be regarded as event-agnostic knowledge without spurious connections with event-specific information. Finally, to balance the convergence speeds across modalities, we dynamically monitor the involvement of each modality during training, through which we could identify the more under-optimized modalities and re-weight the distillation loss accordingly. Our method could be served as a plug-and-play module for existing multimodal fake news detection backbones. Extensive experiments on three public datasets and four state-of-the-art fake news detection backbones show that our proposed method can improve the performance by a large margin.Keywords
Funding Information
- National Natural Science Foundation of China (62141608, 62236010, 62206291)
This publication has 91 references indexed in Scilit:
- Learning rules in spiking neural networks: A surveyNeurocomputing, 2023
- Neural Architecture Search for Spiking Neural Networks2022
- Hybrid spiking neural network for sleep electroencephalogram signalsScience China Information Sciences, 2022
- Binary imbalanced data classification based on diversity oversampling by generative modelsInformation Sciences, 2022
- Numerical scheme and stability analysis of stochastic Fitzhugh–Nagumo modelResults in Physics, 2022
- Trine: Syslog anomaly detection with three transformer encoders in one generative adversarial networkApplied Intelligence, 2021
- Log-Based Anomaly Detection with Multi-Head Scaled Dot-Product Attention Mechanism2021
- LogAttn: Unsupervised Log Anomaly Detection with an AutoEncoder Based Attention Mechanism2021
- Anomaly Detection from Log Files Using Unsupervised Deep Learning2020
- Generalized leaky integrate-and-fire models classify multiple neuron typesNature Communications, 2018