A Turing Test for Grading in Economics: Detection, Grades, and Student Satisfaction in a Blinded Field Experiment
Authors:
- Scott ShaneSSScott Shane
- Roman M. SheremetaRSRoman M. Sheremeta
- Sining WangSWSining Wang
- Heyu XiongHXHeyu Xiong
Preprint
- 11 October 2026
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
Writing assignments are valuable in economics courses but costly to grade, and large language models can now grade written work and draft feedback cheaply. In a semester-long blinded field experiment in two undergraduate economics courses, 37 students completed 24 short written assignments (846 submissions). Three AI graders and three human teaching assistants independently graded every submission on the same five-dimension rubric. Students saw one randomly selected grade and its feedback, without being told its source, then rated their satisfaction and guessed the grader type. AI feedback was easy to tell apart from human feedback: it was more than twice as long and hedged more than seven times as often, and a classifier identified its source with 99.4% accuracy. Students could not: they named the grader type correctly in 52.1% of cases, which is chance (p = 0.229), and did not improve over the semester. The human graders graded 0.68 points higher on a five-point scale, a difference in grading standards whose direction the design does not identify. Human-graded submissions received higher satisfaction ratings, but holding the grade fixed cut the coefficient on the actual grader to nearly zero, and a higher grade raised satisfaction equally from either grader type. Students who believed a human had graded them reported higher satisfaction whichever grader actually had. Instructors can close the grade gap by calibrating grading standards; because satisfaction is associated with the believed rather than the actual grader, disclosure may matter through beliefs.Keywords
This publication has 85 references indexed in Scilit:
- AI or Human? Evaluating Student Feedback Perceptions in Higher Education2024
- Ruled by robots: preference for algorithmic decision makers and perceptions of their choicesPublic Choice, 2024
- Algorithmic legitimacy in clinical decision-makingEthics and Information Technology, 2023
- Applicants’ Fairness Perceptions of Algorithm-Driven Hiring ProceduresJournal of Business Ethics, 2023
- Discourses of artificial intelligence in higher education: a critical literature reviewHigher Education, 2022
- AI Decision Making with Dignity? Contrasting Workers’ Justice Perceptions of Human and AI Decision Making in a Human Resource Management ContextInformation Systems Frontiers, 2022
- Uncertainty, risk and the use of algorithms in policy decisions: a case study on criminal justice in the USAPolicy Sciences, 2021
- Watch Me Improve—Algorithm Aversion and Demonstrating the Ability to LearnBusiness & Information Systems Engineering, 2020
- Fair, Transparent, and Accountable Algorithmic Decision-making ProcessesPhilosophy & Technology, 2017
- What’s in a Name: Exposing Gender Bias in Student Ratings of TeachingInnovative Higher Education, 2014