Evaluating the evaluators: a comparative study of AI and Teacher Assessments in Higher Education
Authors:
- Tugra Karademir CoskunTCTugra Karademir Coskun
- Tugra Karademir CoskunTCTugra Karademir Coskun
- Ayfer AlperAAAyfer Alper
- Ayfer AlperAAAyfer Alper
Open Access
- 1 July 2024
- journal article
- research article
- Published by Edicions de la Universitat de Barcelona in Digital Education Review
- No. 45,p. 124-140
Abstract
This study aims to examine the potential differences between teacher evaluations and artificial intelligence (AI) tool-based assessment systems in university examinations. The research has evaluated a wide spectrum of exams including numerical and verbal course exams, exams with different assessment styles (project, test exam, traditional exam), and both theoretical and practical course exams. These exams were selected using a criterion sampling method and were analyzed using Bland-Altman Analysis and Intraclass Correlation Coefficient (ICC) analyses to assess how AI and teacher evaluations performed across a broad range. The research findings indicate that while there is a high level of proficiency between the total exam scores assessed by artificial intelligence and teacher evaluations; medium consistency was found in the evaluation of visually-based exams, low consistency in video exams, high consistency in test exams, and low consistency in traditional exams. This research is crucial as it helps to identify specific areas where artificial intelligence can either complement or needs improvement in educational assessment, guiding the development of more accurate and fair evaluation tools.Keywords
This publication has 56 references indexed in Scilit:
- AI and formative assessment: The train has left the stationJournal of Research in Science Teaching, 2023
- A Survey of Current Machine Learning Approaches to Student Free-Text Evaluation for Intelligent TutoringInternational Journal of Artificial Intelligence in Education, 2022
- The Promises and Challenges of Artificial Intelligence for Teachers: a Systematic Review of ResearchTechTrends, 2022
- Artificial Intelligence (AI): Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policyInternational Journal of Information Management, 2021
- Authoring Tools for Designing Intelligent Tutoring Systems: a Systematic Review of the LiteratureInternational Journal of Artificial Intelligence in Education, 2018
- When human intelligence meets artificial intelligencePsych Journal, 2018
- Optimization of self‐learning in Computer Engineering courses: An intelligent software system supported by Artificial Neural Network and Vortex Optimization AlgorithmComputer Applications in Engineering Education, 2017
- A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability ResearchJournal of Chiropractic Medicine, 2016
- STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENTThe Lancet, 1986
- Coefficient Alpha and the Internal Structure of TestsPsychometrika, 1951