On Benchmarking Human-like Intelligence in Machines

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Recent advances in Artificial Intelligence (AI) have yielded powerful computational models that, by learning from vast amounts of human-generated data, are increasingly posited as approximate models of human cognition. However, we argue that many current evaluation paradigms for AI are insufficient for assessing human-like cognitive capabilities in these models. We identify a set of key shortcomings: a lack of human-validated labels, inadequate representation of human response variability and uncertainty, and reliance on simplified and ecologically invalid tasks. We support our claims by conducting a human evaluation study on nine existing AI benchmarks, suggesting major limitations in task and label designs. To address these limitations, we propose five concrete recommendations for future AI evaluation efforts that will enable more rigorous and meaningful understanding of human-like cognitive capacities in AI models.
Funding Information
  • Schmidt AI 2050
  • ONR
  • MIT-IBM Watson AI Lab
  • Reid Hoffman and the Siegel Family Foundation
  • King’s College Cambridge
  • Cambridge Trust
  • Turing AI Fellowship (EP/V025279/1)
  • Turing AI Fellowship (EP/V056522/1)
  • Turing AI Fellowship (EP/V056883/1)
  • Leverhulme Trust via CFI

This publication has 83 references indexed in Scilit: