Automatic Grading of Computer Programs: A Machine Learning Approach
Authors:
- Shashank SrikantSSShashank Srikant
- Varun AggarwalVAVarun Aggarwal
- 1 December 2013
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 1, 85-92
Abstract
The automatic evaluation of computer programs is a nascent area of research with a potential for large-scale impact. Extant program assessment systems score mostly based on the number of test-cases passed, providing no insight into the competency of the programmer. In this paper, we present a machine learning framework to automatically grade computer programs. We propose a set of highly-informative features, derived from the abstract representations of a given program, that capture the program's functionality. These features are then used to learn a model to grade the programs, which are built against evaluations done by experts on the basis of a rubric. We show that regression modeling based on the given features provide much better grading than the ubiquitous test-case-pass based grading and rivals the grading accuracy of other open-response problems such as essay grading. We also show that our novel features add significant value over and above basic keyword/expression count features. In addition to this, we propose a novel way of posing computer-program grading as a one-class modeling problem. Our preliminary investigations in the same show promising results and suggest an implicit correlation of our features with the proposed grading-levels (rubric). To the best of the authors' knowledge, this is the first time machine learning has been applied to the problem of grading programs. The work is timely with regard to the recent boom in Massively Online Open Courseware (MOOCs), which promises to produce a significant amount of hand-graded digitized data.Keywords
This publication has 11 references indexed in Scilit:
- Automated feedback generation for introductory programming assignments2013
- Semantic similarity-based grading of student programsInformation and Software Technology, 2007
- Automatic test-based assessment of programmingJournal on Educational Resources in Computing, 2005
- Using testing and JUnit across the curriculumACM SIGCSE Bulletin, 2005
- LOCI: fast outlier detection using the local correlation integral2004
- Web-based peer assessment in learning computer programming2004
- Supporting Students in C++ Programming Courses with Automatic Program Style AssessmentJournal of Information Technology Education: Research, 2004
- Automatic analysis of functional program style2002
- LOFACM SIGMOD Record, 2000
- Automatic submission in an evolutionary approach to computer science teachingComputers & Education, 1995