MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems
Authors:
Preprint
Abstract
While automatic dialogue tutors hold great potential in making education personalized and more accessible, research on such systems has been hampered by a lack of sufficiently large and high-quality datasets. Collecting such datasets remains challenging, as recording tutoring sessions raises privacy concerns and crowdsourcing leads to insufficient data quality. To address this, we propose a framework to generate such dialogues by pairing human teachers with a Large Language Model (LLM) prompted to represent common student errors. We describe how we use this framework to collect MathDial, a dataset of 3k one-to-one teacher-student tutoring dialogues grounded in multi-step math reasoning problems. While models like GPT-3 are good problem solvers, they fail at tutoring because they generate factually incorrect feedback or are prone to revealing solutions to students too early. To overcome this, we let teachers provide learning opportunities to students by guiding them using various scaffolding questions according to a taxonomy of teacher moves. We demonstrate MathDial and its extensive annotations can be used to finetune models to be more effective tutors (and not just solvers). We confirm this by automatic and human evaluation, notably in an interactive setting that measures the trade-off between student solving success and telling solutions. The dataset is released publicly.Keywords
All Related Versions
This publication has 35 references indexed in Scilit:
- GPTeach: Interactive TA Training with GPT-based Students2023
- Using cognitive psychology to understand GPT-3Proceedings of the National Academy of Sciences, 2023
- A systematic review of conversational AI in language education: focusing on the collaboration with human teachersJournal of Research on Technology in Education, 2022
- “Can You Clarify What You Said?”: Studying the Impact of Tutee Agents’ Follow-Up Questions on Tutors’ Learning2021
- Sara, the Lecturer: Improving Learning in Online Education with a Scaffolding-Based Conversational Agent2020
- QuizBot2019
- Teacher–Student Dialogue During Classroom Teaching: Does It Really Impact on Student Outcomes?Journal of the Learning Sciences, 2019
- The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring SystemsEducational Psychologist, 2011
- Scaffolding practices that enhance mathematics learningJournal of Mathematics Teacher Education, 2006
- On the cognitive conflict as an instructional strategy for conceptual change: a critical appraisalLearning and Instruction, 2001