Model-Agnostic Method: Exposing Deepfake Using Pixel-Wise Spatial and Temporal Fingerprints
- 8 June 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Big Data
- Vol. 9 (6) , 1496-1509
Abstract
Deepfake poses a serious threat to the reliability of judicial evidence and intellectual property protection. Existing detection methods either blindly utilize deep learning or use biosignal features, but neither considers spatial and temporal relevance of face features. These methods are increasingly unable to resist the growing realism of fake videos and lack generalization. In this paper, we identify a reliable fingerprint through the consistency of AR coefficients and extend the original PPG signal to 3-dimensional fingerprints to effectively detect fake content. Using these reliable fingerprints, we propose a novel model-agnostic method to expose Deepfake by analyzing temporal and spatial faint synthetic signals hidden in portrait videos. Specifically, our method extracts two types of faint information, i.e., PPG features and AR features, which are used as the basis for forensics in temporal and spatial domains, respectively. PPG allows remote estimation of the heart rate in face videos, and irregular heart rate fluctuations expose traces of tampering. AR coefficients reflect pixel-wise correlation and spatial traces of smoothing caused by up-sampling in the process of generating fake faces. Furthermore, we employ two ACBlock-based DenseNets as classifiers. Our method provides state-of-the-art performance on multiple deep forgery datasets and demonstrates better generalization.Keywords
Funding Information
- National Key R&D Program of China (2019YFC1906201)
- National Natural Science Foundation of China (91748122)
- Xianyang Key R&D Program (S2021ZDYF-SF-0739)
- China Scholarship Council
This publication has 54 references indexed in Scilit:
- Long-term recurrent convolutional networks for visual recognition and description2015
- Ambient and Unobtrusive Cardiorespiratory Monitoring TechniquesIEEE Reviews in Biomedical Engineering, 2015
- Motion-Resistant Remote Imaging Photoplethysmography Based on the Optical Properties of SkinIEEE Transactions on Circuits and Systems for Video Technology, 2014
- Improved motion robustness of remote-PPG by using the blood volume pulse signaturePhysiological Measurement, 2014
- Physiological parameter monitoring of drivers based on video data and independent vector analysis2014
- Non-contact video-based vital sign monitoring using ambient light and auto-regressive modelsPhysiological Measurement, 2014
- Robust Pulse Rate From Chrominance-Based rPPGIEEE Transactions on Biomedical Engineering, 2013
- Advancements in Noncontact, Multiparameter Physiological Measurements Using a WebcamIEEE Transactions on Biomedical Engineering, 2010
- Non-contact, automated cardiac pulse measurements using video imaging and blind source separationOptics Express, 2010
- Texture classification and segmentation using multiresolution simultaneous autoregressive modelsPattern Recognition, 1992