Identifying and mitigating the influence of invalid responses in online surveys: a longitudinal case study
Open Access
- 12 August 2025
- journal article
- research article
- Published by Springer Nature in Quality & Quantity
- Vol. 60 (1) , 1903-1921
Abstract
Use of online surveys is common in educational research. It is well known that artificial intelligence (AI) bots are highly active throughout the internet. Yet, the full impact of bots and their influence in online educational research is not fully understood. The following report describes the iterative methodology and results for identifying and removing suspected bot responses used by one research team across three online, national surveys in the United States and how these processes changed over time. Of the 2,116 responses received across the three surveys, 1,596 responses (75.4%) were removed. Of the 1,596 responses removed, 1,070 responses (67.0% of removed responses) were confidently identified as bots by the research team. Strategies for identifying obvious and non-obvious bot responses in a survey and preventing bots through intentional survey design features are discussed. Implications include improving the reliability of survey data analysis and practices for reporting survey data.Keywords
This publication has 25 references indexed in Scilit:
- Bots, scammers, and fraudulent responders: a year of disrupted data collectionEthics & Behavior, 2025
- Payment versus charitable donations to attract agricultural and natural resource survey participationJournal of the Agricultural and Applied Economics Association, 2023
- Battling bots: Experiences and strategies to mitigate fraudulent responses in online surveysApplied Economic Perspectives and Policy, 2023
- ChatGPT: five priorities for researchNature, 2023
- Ensuring survey research data integrity in the era of internet botsQuality & Quantity, 2021
- Mischief-making bots attacked my scientific surveyNature, 2020
- Attitudinal Survey Characteristics Impacting Participant ResponsesThe Journal of Experimental Education, 2019
- Detecting, Preventing, and Responding to “Fraudsters” in Internet Research: Ethics and TradeoffsJournal of Law, Medicine & Ethics, 2015
- Internet, Phone, Mail, and Mixed‐Mode Surveys2014
- Identifying Multiple Submissions in Internet Research: Preserving Data IntegrityAIDS and Behavior, 2008