When AI Evaluates Its Own Work: Validating Learner-Initiated, AI-Generated Physics Practice Problems
Preprint
Abstract
Large language models (LLMs) can now generate physics practice problems in real time, yet the educational value of these items hinges on rapid, reliable post-generation vetting. We investigated which automated checks are both technically feasible and pedagogically meaningful when exercises are produced on demand within a chatbot interface. A cohort of 34 introductory-physics students generated and attempted 543 problems during exam preparation. Each item was labeled by an expert on a wide range of quality attributes and presented to the learners in pairs to record their preference. We then (i) benchmarked three commodity LLMs as "judges" against the expert labels, (ii) quantified which attributes predict student choice via random-forest models, and (iii) triangulated these results with free-form exit surveys. Only a small subset of the original rubric proved necessary to reliably address student preferences either directly or by proxy. The study demonstrates that scalable formative assessment does not require exhaustive scoring: a carefully curated core of structural and learner-visible checks is sufficient to ensure both technical soundness and user appeal. The findings provide a practical blueprint for deploying real-time, AI-generated practice in physics and other quantitative disciplines.Keywords
All Related Versions
- Version 1, 2025-08-05, ArXiv
- Published version: Physical Review Physics Education Research, 22 (1).
This publication has 42 references indexed in Scilit:
- Multilingual performance of a multimodal artificial intelligence system on multisubject physics concept inventoriesPhysical Review Physics Education Research, 2025
- Can ChatGPT pass a physics degree? Making a case for reformation of assessment of undergraduate degreesEuropean Journal of Physics, 2024
- A survey on students’ use of AI at a technical universityDiscover Education, 2024
- ChatGPT as a tool for honing teachers’ Socratic dialogue skillsPhysics Education, 2024
- Personalizing Algebra to Students’ Individual Interests in an Intelligent Tutoring System: Moderators of ImpactInternational Journal of Artificial Intelligence in Education, 2018
- Mastery-style homework exercises in introductory physics courses: Implementation mattersPhysical Review Physics Education Research, 2018
- Central distractors in Force Concept Inventory dataPhysical Review Physics Education Research, 2018
- A review and critique of context-based physics instruction and assessmentEducational Research Review, 2008
- Comparison of student performance using web and paper‐based homework in college‐level physicsJournal of Research in Science Teaching, 2003
- Teaching Introductory PhysicsPhysics Today, 1997