Integrating AI Tutors in a Programming Course
Open Access
- 5 December 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 130-136
Abstract
RAGMan is an LLM-powered tutoring system that can support a variety of course-specific and homework-specific AI tutors. RAGMan leverages Retrieval Augmented Generation (RAG), as well as strict instructions, to ensure the alignment of the AI tutors' responses. By using RAGMan's AI tutors, students receive assistance with their specific homework assignments without directly obtaining solutions, while also having the ability to ask general programming-related questions. RAGMan was deployed as an optional resource in an introductory programming course with an enrollment of 455 students. It was configured as a set of five homework-specific AI tutors. This paper describes the interactions the students had with the AI tutors, the students' feedback, and a comparative grade analysis. Overall, about half of the students engaged with the AI tutors, and the vast majority of the interactions were legitimate homework questions. When students posed questions within the intended scope, the AI tutors delivered accurate responses 98% of the time. Among the students who used AI tutors, 78% reported that the tutors helped their learning. Beyond AI tutors' ability to provide valuable suggestions, students reported appreciating them for fostering a safe learning environment free from judgment.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Comparing the quality of human and ChatGPT feedback of students’ writingLearning and Instruction, 2024
- Performance and exploration of ChatGPT in medical examination, records and education in Chinese: Pave the way for medical AIInternational Journal of Medical Informatics, 2023
- ChatGPT for good? On opportunities and challenges of large language models for educationLearning and Individual Differences, 2023
- Programming Is Hard - Or at Least It Used to Be2023
- Using Large Language Models to Enhance Programming Error Messages2023
- Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book2023
- Interacting with educational chatbots: A systematic reviewEducation and Information Technologies, 2022
- Co-design for a Competency Self-assessment Chatbot and Survey in Science Education2020
- Online python tutor2013
- A Simplified Monte Carlo Significance Test ProcedureJournal of the Royal Statistical Society Series B: Statistical Methodology, 1968