A Theory-Driven Game-Based Assessment of Collaborative Problem-Solving
Open Access
- 1 September 2026
- journal article
- research article
- Published by MDPI AG in Journal of Intelligence
- Vol. 14 (9) , 197
Abstract
Collaborative problem-solving (CPS) is an essential 21st-century competency, yet its valid and ecologically sound assessment remains challenging due to inconsistent frameworks, poorly aligned task design, and over-reliance on behavior frequency counting in scoring. This study developed and provided empirical support for a theory-driven game-based CPS assessment embedded in a real-time strategy game, using a customized Collaborative Siege Task with asymmetric roles and ill-structured problems, to elicit more realistic collaborative interactions. Based on a validated three-facet CPS framework from Chinese cultural context, the task systematically triggers social and cognitive processes of collaboration across five progressive stages. By sampling from the eastern provinces of China, a pilot study with 36 participants confirmed task feasibility and indicator effectiveness, while a formal study with 298 valid participants examined psychometric properties using item response theory and confirmatory factor analysis. Findings supported the hierarchical higher-order factor structure, satisfactory item difficulty, discrimination, reliability, and criterion-related validity against peer assessment, reasoning ability, and personality. The assessment showed no sex, major, or role bias. This study provides a reliable gamified assessment tool for CPS measurement with demonstrated potential for scalability pending further development of automated scoring procedures, offering implications for large-scale evaluation and competency-oriented instruction.Keywords
Funding Information
- 2025 Youth Fund Project for Humanities and Social Sciences Research of the Ministry of Education (25YJC880029)
- Yangzhou University (5005/142080184)
- Yangzhou University (5005/136020503)
- Green Yang Jinfeng Plan of Yangzhou excellent doctoral funding project (5005/137013654)
This publication has 22 references indexed in Scilit:
- Stealth assessments in digital learning environments: Current trends, new directions, and ethical considerationsJournal of Research on Technology in Education, 2026
- Collaboration in the 21st century: The theory, assessment, and teaching of collaborative problem solvingComputers in Human Behavior, 2020
- Gamified performance assessment of collaborative problem solving skillsComputers in Human Behavior, 2020
- Exploring social and cognitive dimensions of collaborative problem solving in an open online simulation-based taskComputers in Human Behavior, 2020
- Towards a generalized competency model of collaborative problem solvingComputers & Education, 2020
- Innovative Assessment of Collaboration2017
- Measuring problem solving skills via stealth assessment in an engaging video gameComputers in Human Behavior, 2016
- Assessment of Collaborative Problem Solving in Education EnvironmentsApplied Measurement in Education, 2016
- Computer-based Assessment of Collaborative Problem Solving: Exploring the Feasibility of Human-to-Agent ApproachInternational Journal of Artificial Intelligence in Education, 2015
- A very brief measure of the Big-Five personality domainsJournal of Research in Personality, 2003