Using GPT-4 to Augment Unbalanced Data for Automatic Scoring
Authors:
Preprint
Abstract
Machine learning-based automatic scoring can be challenging if students' responses are unbalanced across scoring categories, as it introduces uncertainty in the machine training process. To meet this challenge, we introduce a novel text data augmentation framework using GPT-4, a generative large language model, specifically tailored for unbalanced datasets in automatic scoring. Our experimental dataset comprised student-written responses to two science items. We crafted prompts for GPT-4 to generate responses resembling student-written answers, particularly for the minority scoring classes, to augment the data. We then finetuned DistillBERT for automatic scoring based on the augmented and original datasets. Model performance was assessed using accuracy, precision, recall, and F1 score. We incorporate varied amounts of augmented data to examine scoring performance, and our findings revealed remarkedly improved model performance. The average maximum increase observed across two items is: 3.5% for accuracy, 30.6% for precision, 21.1% for recall, and 24.2% for F1 score. Notably, using just 5% of the augmented data led to substantial improvements: 2.6%, 29.2%, 15.1%, and 19.6%. Interestingly, the extent of improvement varied depending on specific datasets. Moreover, we found that a varying amount of augmented data (5%-40%) was needed to obtain a stable improvement. We also compare models trained with GPT-4 augmented data and those trained with additional student-written responses. The findings indicate that former ones match or even exceed the performance of the latter. Specifically, there is an average difference of 1.7%, 1.9%, 11.0%, and 7.8% for four metrics separately. This research underscores the potential and effectiveness of data augmentation techniques utilizing GPT-4 in addressing unbalanced datasets within automated assessment.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- AugGPT: Leveraging ChatGPT for Text Data AugmentationIEEE Transactions on Big Data, 2025
- Effectiveness of data augmentation to predict students at risk using deep learning algorithmsSocial Network Analysis and Mining, 2023
- Summary of ChatGPT-Related research and perspective towards the future of large language modelsMeta-Radiology, 2023
- Automated Assessment of Student Hand Drawings in Free-Response Items on the Particulate Nature of MatterJournal of Science Education and Technology, 2023
- Data augmentation approaches in natural language processing: A surveyAI Open, 2022
- Using Fair AI with Debiased Network Embeddings to Support Help Seeking in an Online Math Learning Platform2021
- Review of Classification Methods on Unbalanced Data SetsIEEE Access, 2021
- A Meta-Analysis of Machine Learning-Based Science Assessments: Factors Impacting Machine-Human Score AgreementsJournal of Science Education and Technology, 2020
- Applying machine learning in science assessment: a systematic reviewStudies in Science Education, 2020
- Data augmentation for improving deep learning in image classification problem2018