A Fast and Accurate Machine Learning Autograder for the Breakout Assignment
Open Access
- 7 March 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 736-742
Abstract
In this paper, we detail the successful deployment of a machine learning autograder that significantly decreases the grading labor required in the Breakout computer science assignment. This assignment - which tasks students with programming a game consisting of a controllable paddle and a ball that bounces off the paddle to break bricks - is popular for engaging students with introductory computer science concepts, but creates a large grading burden. Due to the game's interactive nature, grading defies traditional unit tests and instead typically requires 8+ minutes of manually playing each student's game to search for bugs. This amounts to 45+ hours of grading in a standard course offering and prevents further widespread adoption of the assignment. Our autograder alleviates this burden by playing each student's game with a reinforcement learning agent and providing videos of discovered bugs to instructors. In an A/B test with manual grading, we find that our human-in-the-loop AI autograder reduces grading time by 44%, while slightly improving grading accuracy by 6%, ultimately saving roughly 30 hours over our deployment in two offerings of the assignment. Our results further suggest the practicality of grading other interactive assignments (e.g., other games or building websites) via similar machine learning techniques. Live demo at https://ezliu.github.io/breakoutgrader.Keywords
Funding Information
- National Science Foundation (DGE-1656518)
- Stanford University (Hoffman Yee Award)
- Intel Foundation
- CIFAR Foundation
This publication has 22 references indexed in Scilit:
- A Summer Introductory Programming Course with Diversity Awareness2023
- Towards understanding the effective design of automated formative feedback for programming assignmentsComputer Science Education, 2021
- Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming TutorInternational Journal of Artificial Intelligence in Education, 2017
- Automated feedback generation for introductory programming assignments2013
- Five Major Shifts in 100 Years of Engineering EducationProceedings of the IEEE, 2012
- EngagementACM SIGCSE Bulletin, 2009
- Engaging students through mobile game developmentACM SIGCSE Bulletin, 2009
- Girls, computer science, and gamesACM SIGCSE Bulletin, 2008
- A games first approach to teaching introductory programming2007
- Nifty assignments2006