The performance of large language models on quantitative and verbal ability tests: Initial evidence and implications for unproctored high‐stakes testing
Open Access
- 17 May 2024
- journal article
- research article
- Published by Wiley in International Journal of Selection and Assessment
- Vol. 32 (4) , 499-511
Abstract
Unproctored assessments are widely used in pre‐employment assessment. However, widely accessible large language models (LLMs) pose challenges for unproctored personnel assessments, given that applicants may use them to artificially inflate their scores beyond their true abilities. This may be particularly concerning in cognitive ability tests, which are widely used and traditionally considered to be less fakeable by humans than personality tests. Thus, this study compares the performance of LLMs on two common types of cognitive tests: quantitative ability (number series completion) and verbal ability (use a passage of text to determine whether a statement is true). The tests investigated are used in real‐world, high‐stakes selection. We also examine the performance of the LLMs across different test formats (i.e., open‐ended vs. multiple choice). Further, we contrast the performance of two LLMs (Generative Pretrained Transformers, GPT‐3.5 and GPT‐4) across multiple prompt approaches and “temperature” settings (i.e., a parameter that determines the amount of randomness in the model's output). We found that the LLMs performed well on the verbal ability test but extremely poorly on the quantitative ability test, even when accounting for the test format. GPT‐4 outperformed GPT‐3.5 across both types of tests. Notably, although prompt approaches and temperature settings did affect LLM test performance, those effects were mostly minor relative to differences across tests and language models. We provide recommendations for securing pre‐employment testing against LLM influences. Additionally, we call for rigorous research investigating the prevalence of LLM usage in pre‐employment testing as well as on how LLM usage affects selection test validity.Keywords
All Related Versions
This publication has 39 references indexed in Scilit:
- Can a computer outfake a human?Personality and Individual Differences, 2023
- ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scopeInternet of Things and Cyber-Physical Systems, 2023
- GPT-3: Its Nature, Scope, Limits, and ConsequencesMinds and Machines, 2020
- A Meta-Analysis of Test Scores in Proctored and Unproctored Ability AssessmentsEuropean Journal of Psychological Assessment, 2020
- How Secure are Unproctored Pre-Employment Tests? Analysis of Inconsistent Test ScoresJournal of Business and Psychology, 2014
- The international cognitive ability resource: Development and initial validation of a public-domain measureIntelligence, 2014
- Personnel selection as a signaling game.Journal of Applied Psychology, 2012
- Internet testing: Equivalence between proctored lab and unproctored field conditionsComputers in Human Behavior, 2008
- Measuring faking in the employment interview: Development and validation of an interview faking behavior scale.Journal of Applied Psychology, 2007
- The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings.Psychological Bulletin, 1998