Assessing a combined human coding and natural language processing method for qualitative analysis in physics education research
Open Access
- 28 October 2025
- conference paper
- Published by American Association of Physics Teachers (AAPT)
- p. 112-117
Abstract
As artificial intelligence becomes increasingly integrated into education research, it is critical to assess the capabilities and limitations of such tools in complex disciplinary contexts. This study explores the effectiveness of natural language processing (NLP) for topic identification using the BERTopic package to analyze undergraduate quantum mechanics syllabi. By comparing AI-generated topics with human-coded themes from a corpus of syllabi from 50 US institutions, we find that while NLP reliably identifies broad and structural elements of syllabi (e.g., policies, textbook references), it is limited in its ability to capture the nuanced and context-dependent content typical of upper-level physics instruction. The AI-derived topics were often underrepresented and diverged from expert human interpretation, particularly in areas requiring deep disciplinary knowledge. Our findings support the use of NLP as a complementary tool for theme generation and cross-validation, but emphasize the continued necessity of expert human analysis for rigorous and meaningful educational research.Keywords
This publication has 6 references indexed in Scilit:
- Modern physics courses: Understanding the content taught in the U.S.Physical Review Physics Education Research, 2025
- Quantum mechanics curriculum in the U.S.: Quantifying the instructional time, content taught, and paradigms usedPhysical Review Physics Education Research, 2025
- Integrating artificial intelligence-based methods into qualitative research in physics education research: A case for computational grounded theoryPhysical Review Physics Education Research, 2023
- Could an artificial-intelligence agent pass an introductory physics course?Physical Review Physics Education Research, 2023
- Classification of open-ended responses to a research-based assessment using natural language processingPhysical Review Physics Education Research, 2022
- Thematic analysis of 18 years of physics education research conference proceedings using natural language processingPhysical Review Physics Education Research, 2020