Do You Really Mean That? Content Driven Audio-Visual Deepfake Dataset and Multimodal Method for Temporal Forgery Localization
Authors:
- 30 November 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-10
Abstract
Due to its high societal impact, deepfake detection is getting active attention in the computer vision community. Most deepfake detection methods rely on identity, facial attributes, and adversarial perturbation-based spatio-temporal modifications at the whole video or random locations while keeping the meaning of the content intact. However, a sophisticated deepfake may contain only a small segment of video/audio manipulation, through which the meaning of the content can be, for example, completely inverted from a sentiment perspective. We introduce a content-driven audio-visual deepfake dataset, termed Localized Audio Visual DeepFake (LAV-DF), explicitly designed for the task of learning temporal forgery localization. Specifically, the content-driven audio-visual manipulations are performed strategically to change the sentiment polarity of the whole video. Our baseline method for benchmarking the proposed dataset is a 3DCNN model, termed as Boundary Aware Temporal Forgery Detection (BA-TFD), which is guided via contrastive, boundary matching, and frame classification loss functions. Our extensive quantitative and qualitative analysis demonstrates the proposed method's strong performance for temporal forgery localization and deepfake detection tasks.Keywords
All Related Versions
This publication has 51 references indexed in Scilit:
- Single Shot Temporal Action Detection2017
- Tacotron: Towards End-to-End Speech Synthesis2017
- CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos2017
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Out of Time: Automated Lip Sync in the Wild2017
- The THUMOS challenge on action recognition for videos “in the wild”Computer Vision and Image Understanding, 2017
- Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs2016
- ActivityNet: A large-scale video benchmark for human activity understanding2015
- Multi-Region Probabilistic Histograms for Robust and Scalable Identity Inference2009
- Learning Internal Representations by Error Propagation1985