Evaluating IBM’s Watson natural language processing artificial intelligence as a short-answer categorization tool for physics education research
Open Access
- 22 March 2024
- journal article
- research article
- Published by American Physical Society (APS) in Physical Review Physics Education Research
- Vol. 20 (1) , 010116
Abstract
Recent advances in publicly available natural language processors (NLP) may enhance the efficiency of analyzing student short-answer responses in physics education research (PER). We train a state-of-the-art NLP, IBM’s Watson, and test its agreement with human coders using two different studies that gathered text responses in which students explain their reasoning on physics-related questions. The first study analyzes 479 student responses to a lab data analysis question and categorizes them by main idea. The second study analyzes 732 student answers to identify the presence or absence of each of the two conceptual themes. When training Watson with approximately one-third to half of the samples, we find that samples labeled with high confidence scores have similar accuracy to human agreement; yet for lower confidence scores, humans outperform the NLP’s labeling accuracy. In addition to studying Watson’s overall accuracy, we use this analysis to better understand factors that impact how Watson categorizes. Using the data from the categorization study, we find that Watson’s algorithm does not appear to be impacted by the disproportionate representation of categories in the training set, and we examine mislabeled statements to identify vocabulary and phrasing that may increase the rate of false positives. Based on this work, we find that, with careful consideration of the research study design and an awareness of the NLP’s limitations, Watson may present a useful tool for large-scale PER studies or classroom analysis tools. Published by the American Physical Society 2024Keywords
Funding Information
- National Science Foundation (DUE 2021099)
This publication has 16 references indexed in Scilit:
- Classification of open-ended responses to a research-based assessment using natural language processingPhysical Review Physics Education Research, 2022
- A Meta-Analysis of Machine Learning-Based Science Assessments: Factors Impacting Machine-Human Score AgreementsJournal of Science Education and Technology, 2020
- Machine learning algorithm for grading open-ended physics questions in TurkishEducation and Information Technologies, 2020
- Applying machine learning in science assessment: a systematic reviewStudies in Science Education, 2020
- Automatic short answer grading and feedback using text mining methodsProcedia Computer Science, 2020
- Validation of automated scoring of science assessmentsJournal of Research in Science Teaching, 2016
- The Eras and Trends of Automatic Short Answer GradingInternational Journal of Artificial Intelligence in Education, 2015
- Transforming Biology Assessment with Machine Learning: Automated Scoring of Written Evolutionary ExplanationsJournal of Science Education and Technology, 2011
- A comparison of human and computer marking of short free-text student responsesComputers & Education, 2010
- Three models for the description of languageIEEE Transactions on Information Theory, 1956