Improving Assessment of Tutoring Practices using Retrieval-Augmented Generation
Preprint
Abstract
One-on-one tutoring is an effective instructional method for enhancing learning, yet its efficacy hinges on tutor competencies. Novice math tutors often prioritize content-specific guidance, neglecting aspects such as social-emotional learning. Social-emotional learning promotes equity and inclusion and nurturing relationships with students, which is crucial for holistic student development. Assessing the competencies of tutors accurately and efficiently can drive the development of tailored tutor training programs. However, evaluating novice tutor ability during real-time tutoring remains challenging as it typically requires experts-in-the-loop. To address this challenge, this preliminary study aims to harness Generative Pre-trained Transformers (GPT), such as GPT-3.5 and GPT-4 models, to automatically assess tutors' ability of using social-emotional tutoring strategies. Moreover, this study also reports on the financial dimensions and considerations of employing these models in real-time and at scale for automated assessment. The current study examined four prompting strategies: two basic Zero-shot prompt strategies, Tree of Thought prompt, and Retrieval-Augmented Generator (RAG) based prompt. The results indicate that the RAG prompt demonstrated more accurate performance (assessed by the level of hallucination and correctness in the generated assessment texts) and lower financial costs than the other strategies evaluated. These findings inform the development of personalized tutor training interventions to enhance the the educational effectiveness of tutored learning.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Evolution Through Large Models2023
- On the role of politeness in online human–human tutoringBritish Journal of Educational Technology, 2023
- Survey of Hallucination in Natural Language GenerationACM Computing Surveys, 2023
- Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction2023
- Educational Equity Through Combined Human-AI Personalization: A Propensity Matching Evaluation2022
- Is it a good move? Mining effective tutoring strategies from human–human tutorial dialoguesFuture Generation Computer Systems, 2022
- The Impressive Effects of Tutoring on PreK-12 Learning: A Systematic Review and Meta-Analysis of the Experimental Evidence2020
- Student and tutor perceptions of effective tutoring in distance educationDistance Education, 2009