Testing the validity of embedding-based similarity and clustering for handwritten physics solutions
Open Access
- 21 October 2026
- journal article
- research article
- Published by American Physical Society (APS) in Physical Review Physics Education Research
- Vol. 22 (2) , 020147
Abstract
Text embeddings are increasingly used in physics education research to organize, compare, and cluster large collections of written text. Their appeal is clear: once student responses have been mapped into a vector space, similarity comparisons and clustering become computationally inexpensive. However, in assessment contexts, the relevant question is not merely whether clusters can be produced but whether the geometry of the embedding space preserves grading-relevant distinctions. We tested this premise using 992 handwritten student-problem solutions from a high-stakes engineering thermodynamics exam, transcribed into five textual representations and embedded using nine embedding mechanisms. We compared embedding similarity and embedding-based hierarchical clusters against human-assigned scores. Across models, representations, and clustering choices, embedding similarity showed a consistent but modest relationship to score similarity, and the resulting clusters were score-enriched but not score-equivalent. Experiments with a synthetic dataset suggest that this may be due to embeddings behaving like novices when categorizing physics-problem solutions; that is, their similarity geometry is strongly influenced by surface features rather than conceptual, semantic structure. These findings suggest that rubric-blind transcription and out-of-the-box, state-of-the-art embeddings can support exploratory organization and human-in-the-loop review of physics solutions, but they do not provide an unsupervised basis for grading without external validation against the assessment construct of interest.Keywords
This publication has 84 references indexed in Scilit:
- Applying Machine Learning in Science Education Research2025
- Mathematical Formulas-Based Scientific Literature Retrieval with Heterogeneous Network Semantic Enhancement2024
- Exploring new depths: Applying machine learning for the analysis of student argumentation in chemistryJournal of Research in Science Teaching, 2023
- Modeling Unproductive Behavior in Online Homework in Terms of Latent Student Traits: An Approach Based on Item Response TheoryJournal of Science Education and Technology, 2016
- The Psychometric Properties of Classroom Response System Data: A Case StudyJournal of Science Education and Technology, 2016
- Hesitant Fuzzy Sets Theory2014
- Early Identification of Student Performance and Effort Using an Online Homework System: A Pilot StudyJournal of Science Education and Technology, 2012
- Modelling Mathematical Reasoning in Physics EducationScience & Education, 2011
- The Andes Physics Tutoring System: An Experiment in Freedom2010
- Graph drawing by force‐directed placementSoftware: Practice and Experience, 1991