ChatGPT versus human essayists: an exploration of the impact of artificial intelligence for authorship and academic integrity in the humanities
Top Cited Papers
Open Access
- 21 October 2024
- journal article
- research article
- Published by Springer Nature in International Journal for Educational Integrity
- Vol. 20 (1) , 1-19
Abstract
Generative AI has prompted educators to reevaluate traditional teaching and assessment methods. This study examines AI’s ability to write essays analysing Old English poetry; human markers assessed and attempted to distinguish them from authentic analyses of poetry by first-year undergraduate students in English at the University of Oxford. Using the standard UK University grading system, AI-written essays averaged a score of 60.46, whilst human essays achieved 63.57, a margin of difference not statistically significant (p = 0.10). Notably, student submissions applied a nuanced understanding of cultural context and secondary criticism to their close reading, while AI essays often described rather than analysed, lacking depth in the evaluation of poetic features, and sometimes failing to properly recognise key aspects of passages. Distinguishing features of human essays included detailed and sustained analysis of poetic style, as well as spelling errors and lack of structural cohesion. AI essays, on the other hand, exhibited a more formal structure and tone but sometimes fell short in incisive critique of poetic form and effect. Human markers correctly identified the origin of essays 79.41% of the time. Additionally, we compare three purported AI detectors, finding that the best, ‘Quillbot’, correctly identified the origin of essays 95.59% of the time. However, given the high threshold for academic misconduct, conclusively determining origin remains challenging. The research also highlights the potential benefits of generative AI’s ability to advise on structuring essays and suggesting avenues for research. We advocate for transparency regarding AI’s capabilities and limitations, and this study underscores the importance of human critical engagement in teaching and learning in Higher Education. As AI’s proficiency grows, educators must reevaluate what authentic assessment is, and consider implementing dynamic, holistic methods to ensure academic integrity.Keywords
This publication has 27 references indexed in Scilit:
- Generative AI detection in higher education assessmentsNew Directions for Teaching and Learning, 2024
- ChatGPT for good? On opportunities and challenges of large language models for educationLearning and Individual Differences, 2023
- Automated Essay Writing: An AIED OpinionInternational Journal of Artificial Intelligence in Education, 2022
- The Promises and Challenges of Artificial Intelligence for Teachers: a Systematic Review of ResearchTechTrends, 2022
- Rethinking authentic assessment: work, well-being, and societyHigher Education, 2022
- Artificial intelligence versus Maya Angelou: Experimental evidence that people cannot differentiate AI-generated from human-written poetryComputers in Human Behavior, 2021
- University students’ intention to use search engines for research purposes: A structural equation modeling approachEducation and Information Technologies, 2019
- Unpacking teachers’ intentions to integrate technology: A meta-analysisEducational Research Review, 2019
- Automated Assessment of Non-Native Learner Essays: Investigating the Role of Linguistic FeaturesInternational Journal of Artificial Intelligence in Education, 2017
- State-of-the-art automated essay scoring: Competition, results, and future directions from a United States demonstrationAssessing Writing, 2014