A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
Authors:
- Jacob DoughtyJDJacob Doughty
- Zipiao WanZWZipiao Wan
- Anishka BompelliABAnishka Bompelli
- Jubahed QayumJQJubahed Qayum
- Taozhi WangTWTaozhi Wang
- Juran ZhangJZJuran Zhang
- Yujia ZhengYZYujia Zheng
- Aidan DoyleADAidan Doyle
- Pragnya SridharPSPragnya Sridhar
- Arav AgarwalAAArav Agarwal
- Christopher BogartCBChristopher Bogart
- Eric KeylorEKEric Keylor
Preprint
Abstract
There is a constant need for educators to develop and maintain effective up-to-date assessments. While there is a growing body of research in computing education on utilizing large language models (LLMs) in generation and engagement with coding exercises, the use of LLMs for generating programming MCQs has not been extensively explored. We analyzed the capability of GPT-4 to produce multiple-choice questions (MCQs) aligned with specific learning objectives (LOs) from Python programming classes in higher education. Specifically, we developed an LLM-powered (GPT-4) system for generation of MCQs from high-level course context and module-level LOs. We evaluated 651 LLM-generated and 449 human-crafted MCQs aligned to 246 LOs from 6 Python courses. We found that GPT-4 was capable of producing MCQs with clear language, a single correct choice, and high-quality distractors. We also observed that the generated MCQs appeared to be well-aligned with the LOs. Our findings can be leveraged by educators wishing to take advantage of the state-of-the-art generative models to support MCQ authoring efforts.Keywords
All Related Versions
- Version 1, 2023-12-05, ArXiv
- Published version: , 114.
This publication has 23 references indexed in Scilit:
- From GPT-3 to GPT-4: On the Evolving Efficacy of LLMs to Answer Multiple-Choice Questions for Programming Classes in Higher Education2024
- Generating Multiple Choice Questions for Computing Courses Using Large Language Models2023
- Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language2023
- Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book2023
- Generating Diverse Code Explanations using the GPT-3 Large Language Model2022
- Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models2022
- End-to-End generation of Multiple-Choice questions using Text-to-Text transfer Transformer modelsExpert Systems with Applications, 2022
- A Systematic Review of Automatic Question Generation for Educational PurposesInternational Journal of Artificial Intelligence in Education, 2019
- Automatic Multiple Choice Question Generation From Text: A SurveyIEEE Transactions on Learning Technologies, 2018
- Guide To Developing High-Quality, Reliable, and Valid Multiple-Choice AssessmentsJournal of Chemical Education, 2014