ChatGPT and Cheat Detection in CS1 Using a Program Autograding System
Open Access
- 3 July 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 367-373
Abstract
We experimented with ChatGPT's ability to write programs in a CS1 class, and the ability of a popular tool to auto-detect ChatGPT-written programs. We found ChatGPT was proficient at generating correct programs from a mere copy-paste of the English programming assignment specifications. However, running ChatGPT for 10 programming assignments and acting as 20 different students, and using zyBook's APEX beta tool for academic integrity, we found: (1) ChatGPT-generated programs tend to use a programming style departing from the style taught in the textbook or by the instructor, and these "style anomalies" were automatically detected. (2) Although ChatGPT may for the same assignment generate a few different program solutions for different students, ChatGPT often generates highly-similar programs for different students, so if enough students in a class (e.g., 5 or more) use ChatGPT, their programs will likely be flagged by a similarity checker. (3) If students are required to do all programming in the autograder's IDE, then a student using ChatGPT ends up showing very little time relative to classmates, which is automatically flagged. (4) Manually, we observed that if a student consistently uses ChatGPT to submit programs, the programming style may vary across programs, something normal students don't do; automation of style inconsistency detection was recently added to APEX. In short, while there will no doubt be an arms race between AI-generated programs and automatic detection of AI-generated programs, currently students using ChatGPT for multiple CS1 programs can be detected by automated tools such as zyBooks' APEX.Keywords
Funding Information
- National Science Foundation (2111323, 2313793)
This publication has 13 references indexed in Scilit:
- ChatGPT for Teaching and Learning: An Experience from Data Science Education2023
- Variability-Inducing Requirements for Programs: Increasing Solution Variability for Similarity Checking2023
- ChatGPT for Education and Research: Opportunities, Threats, and StrategiesApplied Sciences, 2023
- “We Need To Talk About ChatGPT”: The Future of AI and Higher Education2023
- Chatting and cheating: Ensuring academic integrity in the era of ChatGPTInnovations in Education and Teaching International, 2023
- Impact of Several Low-Effort Cheating-Reduction Methods in a CS1 Class2023
- Engineering Education in the Era of ChatGPT: Promise and Pitfalls of Generative AI for EducationTechRxiv, 2022
- Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewersbioRxiv, 2022
- The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming2022
- Investigating Student Plagiarism Patterns and Correlations to Grades2017