MoE-FFD: Mixture of Experts for Generalized and Parameter-Efficient Face Forgery Detection
Top Cited Papers
- 29 August 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Dependable and Secure Computing
- Vol. 23 (1) , 82-96
Abstract
Deepfakes have recently raised significant trust issues and security concerns among the public. Compared to CNN-based face forgery detectors, ViT-based methods take advantage of the expressivity of transformers, achieving superior detection performance. However, these approaches still exhibit the following limitations: (1) Fully fine-tuning ViT-based models from ImageNet weights demands substantial computational and storage resources; (2) ViT-based methods struggle to capture local forgery clues, leading to model bias; (3) These methods limit their scope on only one or few face forgery features, resulting in limited generalizability. To tackle these challenges, this work introduces Mixture-of-Experts modules for Face Forgery Detection (MoE-FFD), a generalized yet parameter-efficient ViT-based approach. MoE-FFD only updates lightweight Low-Rank Adaptation (LoRA) and Adapter layers while keeping the ViT backbone frozen, thereby achieving parameter-efficient training. Moreover, MoE-FFD leverages the expressivity of transformers and local priors of CNNs to simultaneously extract global and local forgery clues. Additionally, novel MoE modules are designed to scale the model's capacity and smartly select optimal forgery experts, further enhancing forgery detection performance. Our proposed learning scheme can be seamlessly adapted to various transformer backbones in a plug-and-play manner. Extensive experimental results demonstrate that the proposed method achieves state-of-the-art face forgery detection performance with significantly reduced parameter overhead in cross-dataset, cross-manipulation, and robustness evaluations. Our ablation studies further validate the effectiveness of the designed components and the proposed learning scheme. The code is available at: https://github.com/LoveSiameseCat/MoE-FFD.Keywords
All Related Versions
Funding Information
- OTS Research Programme (S24T2TS006)
- National Natural Science Foundation of China (62502187)
- Natural Science Foundation of Jiangxi Province of China (20252BAC240015)
- Early Career Scheme (21200522)
This publication has 66 references indexed in Scilit:
- UIA-ViT: Unsupervised Inconsistency-Aware Method Based on Vision Transformer for Face Forgery Detection2022
- Detecting and Recovering Sequential DeepFake Manipulation2022
- Visual Prompt Tuning2022
- Two-Branch Recurrent Network for Isolating Deepfakes in Videos2020
- Thinking in Frequency: Face Forgery Detection by Mining Frequency-Aware Clues2020
- Learning in the Frequency Domain2020
- DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection2020
- Xception: Deep Learning with Depthwise Separable Convolutions2017
- Face2Face: Real-Time Face Capture and Reenactment of RGB Videos2016
- ImageNet: A large-scale hierarchical image database2009