Grading Machines: Can AI Exam-Grading Replace Law Professors?
Open Access
- 8 April 2026
- journal article
- Published by SAGE Publications in Journal of Law & Empirical Analysis
- Vol. 3 (1) , 2-22
Abstract
In the past few years, large language models (LLMs) have achieved significant technical advances, enabling legal-advocacy organizations to adopt them as complements to—or substitutes for—lawyers and other human experts. The role of LLMs in legal education, however, is underexplored. While several studies have examined LLMs’ performance in taking law school exams, finding mixed results, there have been no published studies systematically analyzing LLMs’ competence at one of law professors’ chief responsibilities: grading law school exams. This paper presents results of an analysis of how LLMs perform in evaluating student responses to legal analysis questions of the kind typically contained in law school exams. The data come from exams in four subjects administered at top-30 U.S. law schools. Unlike some projects in computer or data science, our goal is not to design a new LLM that minimizes error or that maximizes agreement with human graders. Rather, we seek to determine whether existing models—which can be straightforwardly applied by most professors and students—are already suitable for law exam evaluation. We find that, when provided with a detailed rubric, the LLM grades correlate with the human grader at Pearson correlation coefficients of up to 0.93. Our findings suggest that, even if they do not fully replace humans in the near future, LLMs could soon be put to valuable tasks by law school professors, such as reviewing and validating professor grading, providing substantive feedback on ungraded midterms, and providing students feedback on self-administered practice exams.Keywords
This publication has 37 references indexed in Scilit:
- Large language models in cryptocurrency securities cases: can a GPT model meaningfully assist lawyers?Artificial Intelligence and Law, 2024
- Re-evaluating GPT-4’s bar exam performanceArtificial Intelligence and Law, 2024
- The risk ethics of autonomous vehicles: an empirical approachScientific Reports, 2024
- How subjective CT image quality assessment becomes surprisingly reliable: pairwise comparisons instead of Likert scaleEuropean Radiology, 2024
- Summary of ChatGPT-Related research and perspective towards the future of large language modelsMeta-Radiology, 2023
- Improving the signal quality of gradesThe Journal of Law, Economics, and Organization, 2023
- A Pairwise Comparison Framework for Fast, Flexible, and Reliable Human Coding of Political TextsAmerican Political Science Review, 2017
- Human Decisions and Machine PredictionsThe Quarterly Journal of Economics, 2017
- Preventing halo bias in grading the work of university studentsCogent Psychology, 2014
- WRITING ASSESSMENT IN ADMISSION TO HIGHER EDUCATION: REVIEW AND FRAMEWORKETS Research Report Series, 1999