Characteristics and Prevalence of Fake Social Media Profiles with AI-generated Faces
Open Access
- 18 September 2024
- journal article
- Published by Stanford Internet Observatory in Journal of Online Trust and Safety
- Vol. 2 (4)
Abstract
Recent advancements in generative artificial intelligence (AI) have raised concerns about its potential to create convincing fake social media accounts, but empirical evidence is lacking. In this paper, we present a systematic analysis of Twitter (X) accounts using human faces generated by Generative Adversarial Networks (GANs) for their profile pictures. We present a dataset of 1,420 such accounts and show that they are used to spread scams, disseminate spam, and amplify coordinated messages, among other inauthentic activities. Leveraging a feature of GAN-generated faces—consistent eye placement—and supplementing it with human annotation, we devise an effective method for identifying GAN-generated profiles in the wild. Applying this method to a random sample of active Twitter users, we estimate a lower bound for the prevalence of profiles using GAN-generated faces between 0.021% and 0.044%—around 10,000 daily active accounts. These findings underscore the emerging threats posed by multimodal generative AI. We release the source code of our detection method and the data we collect to facilitate further investigation. Additionally, we provide practical heuristics to assist social media users in recognizing such accounts.Keywords
All Related Versions
This publication has 34 references indexed in Scilit:
- The rise and potential of large language model based agents: a surveyScience China Information Sciences, 2025
- Exposure to social bots amplifies perceptual biases and regulation propensityScientific Reports, 2023
- Addressing the harms of AI-generated inauthentic contentNature Machine Intelligence, 2023
- Human heuristics for AI-generated language are flawedProceedings of the National Academy of Sciences of the United States of America, 2023
- Testing human ability to detect ‘deepfake’ images of human facesJournal of Cybersecurity, 2023
- The Emerging Threat of Ai-driven Cyber Attacks: A ReviewApplied Artificial Intelligence, 2022
- AI-synthesized faces are indistinguishable from real faces and more trustworthyProceedings of the National Academy of Sciences of the United States of America, 2022
- GANBOT: a GAN-based framework for social bot detectionSocial Network Analysis and Mining, 2021
- Weaponized AI for cyber attacksJournal of Information Security and Applications, 2021
- Arming the public with artificial intelligence to counter social botsHuman Behavior and Emerging Technologies, 2019