Initial Attempts to Detect or Screen Out AI Responses Prove Elusive in the Age of Agentic AI
- 1 January 2026
- journal article
- Published by Wiley in International Journal of Eating Disorders
- Vol. 59 (4) , 666-673
Abstract
Objective: Although screening for bots and/or using costly panel services for recruiting participants online has become increasingly necessary, such efforts may no longer ensure the validity of data collected online. Newly released agentic AI models, such as the ChatGPT agent, have the ability to complete surveys relatively indistinguishably from humans. Methods: The current paper outlines efforts that the body image, weight, and eating disorders (BIWED) lab has undergone to screen for and detect AI data completion reliably and validly. Results: There are some tasks that ChatGPT agents do not perform identically to human responders (e.g., video tasks, online games, open‐ended responses, and reCAPTCHA). We present the methods that have been the most successful at identifying AI agent survey completion. Discussion: We discuss potential solutions, field‐wide concerns, and future directions for the field more broadly.Keywords
This publication has 15 references indexed in Scilit:
- Yes stormtrooper, these are the droids you are looking for: Identifying and preliminarily evaluating bot and fraud detection strategies in online psychological research.Psychological Methods, 2025
- The potential existential threat of large language models to online survey researchProceedings of the National Academy of Sciences, 2025
- Trust in climate scientists is associated with political ideology: A 26-country analysisJournal of Environmental Psychology, 2025
- A quasi-experimental study examining the efficacy of multimodal bot screening tools and recommendations to preserve data integrity in online psychological research.American Psychologist, 2024
- Battling bots: Experiences and strategies to mitigate fraudulent responses in online surveysApplied Economic Perspectives and Policy, 2023
- The Comprehensive Infrequency/Frequency Item Repository (CIFR): An online database of items for detecting careless/insufficient-effort responders in survey dataPersonality and Individual Differences, 2023
- Eating disorder related research using Amazon Mechanical Turk (MTurk): Friend or foe?: Commentary on Burnette et al. (2021)International Journal of Eating Disorders, 2022
- Concerns and recommendations for using AmazonMTurkfor eating disorder researchInternational Journal of Eating Disorders, 2021
- Impact of survey length and compensation on validity, reliability, and sample characteristics for Ultrashort-, Short-, and Long-Research Participant Perception SurveysJournal of Clinical and Translational Science, 2018
- Why Most Published Research Findings Are FalsePLOS Medicine, 2005