Leveraging Human Feedback to Scale Educational Datasets: Combining Crowdworkers and Comparative Judgement
Open Access
- 20 July 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 411-415
Abstract
Machine Learning models have many potentially beneficial applications in education settings, but a key barrier to their development is securing enough high-quality, labelled data to train these models. This process has traditionally relied on highly skilled raters using complex, multi-class rubrics, which made labelling expensive and difficult to scale. A more scalable approach would be to use non-expert crowdworkers to evaluate student work, but maintaining high levels of accuracy and inter-rater reliability when using non-expert workers can be challenging. This paper reports on two experiments in which non-expert crowdworkers hired to evaluate (i.e., score) student work and were randomly assigned to one of two conditions: the control, where they were asked to assign a rubrics based score (i.e., a categorical judgement), or the treatment, where they were shown the same student answers, but were asked to decide which of two candidate answers was better (i.e., a comparative/preference-based judgement). We found that using comparative judgement substantially improved inter-rater reliability on both tasks. These results are in-line with well-established literature on the benefits of comparative judgement in the field of educational assessment, as well as with recent trends in artificial intelligence research, where comparative judgement is becoming the preferred method for providing human feedback on model outputs. These results are novel and important in demonstrating the effects of using the combination of comparative judgement and crowdworkers to evaluate educational dataKeywords
All Related Versions
This publication has 12 references indexed in Scilit:
- What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?2021
- Interrater Reliability in Systematic Review Methodology: Exploring Variation in Coder Decision-MakingSociological Methods & Research, 2018
- Responsible research with crowdsCommunications of the ACM, 2018
- Measuring inter-rater reliability for nominal data – which coefficients and confidence intervals are appropriate?BMC Medical Research Methodology, 2016
- Evaluating Comparative Judgment as an Approach to Essay ScoringApplied Measurement in Education, 2016
- How accurate are examiners’ holistic judgements of script quality?Assessment in Education: Principles, Policy & Practice, 2013
- Comparative judgement for assessmentInternational Journal of Technology and Design Education, 2011
- Beyond kappa: A review of interrater agreement measuresCanadian Journal of Statistics, 1999
- Reliability of Content Analysis: The Case of Nominal Scale CodingPublic Opinion Quarterly, 1955
- Group Judgments in the Field of Lifted Weights.Journal of Experimental Psychology, 1924