Evaluating Copilot on CS1 Code Writing Problems with Suppressed Specifications
Open Access
- 9 December 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 104-107
Abstract
Code writing problems in introductory programming (CS1) courses typically ask students to write simple functions or programs based on detailed natural-language specifications. These details can be leveraged by large language models (LLMs), accessible to students via tools such as GitHub Copilot, to generate solutions that are often correct. CS1 instructors who are unwilling or unable to prohibit such usage must consider variants of traditional code writing problems that align with their learning objectives but are more difficult for LLMs to solve. Since LLMs are sensitive to the level of details in their prompts, it is natural to consider variants where details are progressively trimmed from the specifications of traditional code writing problems, and consequent ambiguities are clarified via examples. We consider an extreme variant, where all natural language is suppressed except for meaningful names of functions and their arguments. We evaluate the performance of Copilot on suppressed specification versions of 153 such problems drawn from the CodeCheck repository. If Copilot initially fails to generate a correct solution, we augment each suppressed specification with as few clarifying examples as possible to obtain a correct solution. Copilot solves 134 problems (87%) with just 0.7 examples on average, requiring no examples in 78 instances. Thus, modifying traditional code-writing problems by merely trimming specification details is unlikely to thwart sophisticated LLMs such as GitHub Copilot.Keywords
This publication has 18 references indexed in Scilit:
- Automated Assessment in Computer Science Education: A State-of-the-Art ReviewACM Transactions on Computing Education, 2022
- Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm2021
- First Things First2019
- Who Tests the Testers?2018
- Metacognitive Difficulties Faced by Novice Programmers in Automated Assessment Tools2018
- Examining the Enrollment Growth2017
- The Role of Self-Regulation in Programming Problem Solving Process and Success2016
- Measuring the effect of inventing practice exercises on learning in an introductory programming course2015
- Assessing the assessment of programming abilityACM SIGCSE Bulletin, 2004
- The introductory programming course in computer science1978