On the Robustness of Stealth Assessment
- 27 August 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Games
- Vol. 13 (2) , 180-192
Abstract
Stealth assessment is a methodology that utilizes machine learning for processing unobtrusively collected data from serious games to produce inferences regarding learners' mastery level. Although stealth assessment can produce valid and reliable assessments, its robustness over a wide a range of conditions has not been examined yet. The main reason is its complex, laborious, and time-consuming practical application. Therefore, its exposure to different conditions has been limited, as well as its wider uptake from the serious game community. Nevertheless, a framework for developing a generic tool has been proposed to lift its barriers. In this study, a generic SA software tool was developed based on this framework to examine the robustness of the stealth assessment methodology under various conditions. In specific, the conditions relate to (a) processing datasets of different distribution types and sizes (960 datasets containing a total of 72.336.000 data points are used for this reason), (b) utilizing two different machine learning algorithms (Gaussian Naive Bayes Network and C4.5), and (c) using statistical models relating to two different competency constructs. Results show that stealth assessment is a robust methodology, whilst the generic SA tool is a highly accurate tool capable of handling efficiently a wide range of conditions.Keywords
Funding Information
- RAGE project
This publication has 17 references indexed in Scilit:
- Game-based Learning and 21st century skills: A review of recent researchComputers in Human Behavior, 2016
- An update to the systematic literature review of empirical evidence of the impacts and outcomes of computer games and serious gamesComputers & Education, 2016
- DeepStealth: Leveraging Deep Learning Models for Stealth Assessment in Game-Based Learning Environments2015
- Assessment and Learning of Qualitative Physics in Newton's PlaygroundThe Journal of Educational Research, 2013
- A meta-analysis of the cognitive and motivational effects of serious games.Journal of Educational Psychology, 2013
- Assessment in and of Serious Games: An OverviewAdvances in Human-Computer Interaction, 2013
- A few useful things to know about machine learningCommunications of the ACM, 2012
- The Role of Construct Definition in the Creation of Formative Assessments in Game-Based Learning2012
- A BRIEF INTRODUCTION TO EVIDENCE‐CENTERED DESIGNETS Research Report Series, 2003
- Focus Article: On the Structure of Educational AssessmentsMeasurement: Interdisciplinary Research and Perspectives, 2003