Artificial intelligence in scale development: evaluating AI-generated survey items against gold standard measures
Authors:
Open Access
- 25 August 2025
- journal article
- research article
- Published by Springer Nature in Current Psychology
- Vol. 44 (20) , 16339-16350
Abstract
The rise of artificial intelligence (AI) presents a unique opportunity to enhance research capacity, yet rigorous evaluation of AI in psychological science is essential. This study explores the application of ChatGPT version 3.5 for developing survey items and compares their psychometric properties to those of a validated instrument measuring the construct of grit. In Study 1, an exploratory factor analysis with 180 college students revealed that AI-generated items replicated the two-factor structure of the Short Grit Scale and demonstrated high internal consistency reliability (Factor 1, α = 0.94; Factor 2, α = 0.93) with moderate to strong correlations with the Short Grit Scale. In Study 2, a confirmatory factor analysis with a larger sample of 366 participants confirmed the two-factor structure with strong factor loadings ranging from 0.78 to 0.88 and acceptable model fit indices (CFI = 0.97, TLI = 0.95, RMSEA = 0.09, SRMR = 0.04). Additionally, hierarchical regression analysis showed that AI-generated items predicted academic performance and explained more of the variance in grade point average than the Short Grit Scale. This study provides early insights into AI’s potential to support researchers in the lengthy scale development process.Keywords
Funding Information
- College of Charleston
This publication has 30 references indexed in Scilit:
- Comparing network structures on three aspects: A permutation test.Psychological Methods, 2023
- The Ethics of Psychological Artificial Intelligence: Clinical ConsiderationsCounseling and Values, 2021
- A Psychometric Evaluation of the Short Grit ScaleEuropean Journal of Psychological Assessment, 2020
- Examining the relationship between grit and academic achievement within K‐12 and higher education: A systematic reviewPsychology in the Schools, 2019
- Much ado about grit: A meta-analytic synthesis of the grit literature.Journal of Personality and Social Psychology, 2017
- The Use of Cronbach’s Alpha When Developing and Reporting Research Instruments in Science EducationResearch in Science Education, 2017
- Grit: Perseverance and passion for long-term goals.Journal of Personality and Social Psychology, 2007
- An Evaluation of a Surveillance System for Patients Discharged from the Acute Psychiatry Unit in Southern TasmaniaAustralian & New Zealand Journal of Psychiatry, 1999
- Power analysis and determination of sample size for covariance structure modeling.Psychological Methods, 1996
- Constructing validity: Basic issues in objective scale development.Psychological Assessment, 1995