Evaluating General-Purpose AI with Psychometrics
Top Cited Papers
Open Access
- 24 April 2026
- journal article
- Published by Association for Computing Machinery (ACM) in Communications of the ACM
- Vol. 69 (5) , 92-102
Abstract
Rigorous evaluation of general-purpose AI systems such as large language models should allow for deepened understanding of their capabilities and effective mitigation of their risks. The current evaluation paradigm, mostly reliant on benchmarks aggregating scores on one or more tasks, lacks the scientific machinery for predicting performance on unforeseen tasks and explaining the variability of results. Moreover, existing benchmarks raise growing concerns about their reliability and validity. To tackle these challenges, we vindicate psychometrics, the science of psychological measurement, as a methodology for identifying and measuring constructs that underlie AI performance across multiple tasks. To raise awareness, we first identify the key advantages of adapting psychometric principles to AI evaluation through concrete examples; second, we distinguish sound applications of psychometric techniques from oversimplified ones and warn against common pitfalls; and third, to encourage general use, we introduce a systematic psychometric framework and an operational evaluation pipeline, which provide practical implementation guidance. In the end, we discuss underexplored avenues and societal implications that open new research directions for the use of psychometrics in broader AI research.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- Evidence of interrelated cognitive-like capabilities in large language models: Indications of artificial general intelligence or achievement?Intelligence, 2024
- Role play with large language modelsNature, 2023
- Towards a generalized competency model of collaborative problem solvingComputers & Education, 2020
- The Measure of All Minds2017
- IQ tests are not for machines, yetIntelligence, 2012
- Cognitive ability, personality, and academic performance in adolescencePersonality and Individual Differences, 2008
- Chapter 13 Qualitative Spatial Representation and Reasoning2008
- Confabulation Extending across Episodic, Personal, and General Semantic MemoryCognitive Neuropsychology, 1997
- The development of markers for the Big-Five factor structure.Psychological Assessment, 1992
- Resilience and development: Contributions from the study of children who overcome adversityDevelopment and Psychopathology, 1990