Can ChatGPT evaluate research quality?
Author:
Top Cited Papers
Open Access
- 1 May 2024
- journal article
- research article
- Published by Walter de Gruyter GmbH in Journal of Data and Information Science
- Vol. 9 (2) , 1-21
Abstract
Purpose: Assess whether ChatGPT 4.0 is accurate enough to perform research evaluations on journal articles to automate this time-consuming task. Design/methodology/approach: Test the extent to which ChatGPT-4 can assess the quality of journal articles using a case study of the published scoring guidelines of the UK Research Excellence Framework (REF) 2021 to create a research evaluation ChatGPT. This was applied to 51 of my own articles and compared against my own quality judgements. Findings: ChatGPT-4 can produce plausible document summaries and quality evaluation rationales that match the REF criteria. Its overall scores have weak correlations with my self-evaluation scores of the same documents (averaging r=0.281 over 15 iterations, with 8 being statistically significantly different from 0). In contrast, the average scores from the 15 iterations produced a statistically significant positive correlation of 0.509. Thus, averaging scores from multiple ChatGPT-4 rounds seems more effective than individual scores. The positive correlation may be due to ChatGPT being able to extract the author’s significance, rigour, and originality claims from inside each paper. If my weakest articles are removed, then the correlation with average scores (r=0.200) falls below statistical significance, suggesting that ChatGPT struggles to make fine-grained evaluations. Research limitations: The data is self-evaluations of a convenience sample of articles from one academic in one field. Practical implications: Overall, ChatGPT does not yet seem to be accurate enough to be trusted for any formal or informal research quality evaluation tasks. Research evaluators, including journal editors, should therefore take steps to control its use. Originality/value: This is the first published attempt at post-publication expert review accuracy testing for ChatGPT.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Can ChatGPT and Other AI Bots Serve as Peer Reviewers?ACS Energy Letters, 2023
- A comprehensive survey of ChatGPT: Advancements, applications, prospects, and challengesMeta-Radiology, 2023
- Guidance for Authors, Peer Reviewers, and Editors on Use of AI, Language Models, and ChatbotsJAMA, 2023
- Using AI Tools in Writing Peer Review Reports: Should Academic Journals Embrace the Use of ChatGPT?Annals of Biomedical Engineering, 2023
- ChatGPT: Jack of all trades, master of noneInformation Fusion, 2023
- Does the perceived quality of interdisciplinary research vary between fields?Journal of Documentation, 2023
- Reviewer assignment algorithms for peer review automation: A surveyInformation Processing & Management, 2022
- Co-existing Notions of Research Quality: A Framework to Study Context-specific Understandings of Good ResearchMinerva, 2019
- Unique, but still best practice? The Research Excellence Framework (REF) from an international perspectivePalgrave Communications, 2017
- Stat-checking software stirs up psychologyNature, 2016