Evaluating General-Purpose AI with Psychometrics
Preprint
Abstract
Artificial intelligence (AI) has witnessed an evolution from task-specific to general-purpose systems that trend toward human versatility. As AI systems begin to play pivotal roles in society, it is important to ensure that they are adequately evaluated. Current AI benchmarks typically assess performance on collections of specific tasks. This has drawbacks when used for assessing general-purpose AI systems. First, it is difficult to predict whether AI systems could complete a new task it has never seen or that did not previously exist. Second, these benchmarks often focus on overall performance metrics, potentially overlooking the finer details crucial for making informed decisions. Lastly, there are growing concerns about the reliability of existing benchmarks and questions about what is being measured. To solve these challenges, this paper suggests that psychometrics, the science of psychological measurement, should be placed at the core of evaluating general-purpose AI. Psychometrics provides a rigorous methodology for identifying and measuring the latent constructs that underlie performance across multiple tasks. We discuss its merits, warn against potential pitfalls, and propose a framework for putting it into practice. Finally, we explore future opportunities to integrate psychometrics with AI.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- “Minimum Necessary Rigor” in empirically evaluating human–AI work systemsAI Magazine, 2023
- Foundation models for generalist medical artificial intelligenceNature, 2023
- Item response theory in AI: Analysing machine learning classifiers at the instance levelArtificial Intelligence, 2019
- Computer models solving intelligence test problems: Progress and implicationsArtificial Intelligence, 2016
- IQ tests are not for machines, yetIntelligence, 2012
- A meta-analysis of the convergent validity of self-control measuresJournal of Research in Personality, 2011
- Incremental validity of thinking styles in predicting academic achievements: an experimental study in hypermedia learning environmentsEducational Psychology, 2010
- Cognitive ability, personality, and academic performance in adolescencePersonality and Individual Differences, 2008
- Three Multivariate Models: Factor Analysis, Latent Structure Analysis, and Latent Profile AnalysisPsychometrika, 1959
- Construct validity in psychological tests.Psychological Bulletin, 1955