Research quality evaluation by AI in the era of large language models: advantages, disadvantages, and systemic effects – An opinion paper
Author:
Top Cited Papers
Open Access
- 28 July 2025
- journal article
- research article
- Published by Springer Nature in Scientometrics
- Vol. 130 (10) , 5309-5321
Abstract
Artificial Intelligence (AI) technologies like ChatGPT now threaten bibliometrics as the primary generators of research quality indicators. They are already used in at least one research quality evaluation system and evidence suggests that they are used informally by many peer reviewers. Since harnessing bibliometrics to support research evaluation continues to be controversial, this article reviews the corresponding advantages and disadvantages of AI-generated quality scores. From a technical perspective, generative AI based on Large Language Models (LLMs) equals or surpasses bibliometrics in most important dimensions, including accuracy (mostly higher correlations with human scores), and coverage (more fields, more recent years) and may reflect more research quality dimensions. Like bibliometrics, current LLMs do not “measure” research quality, however. On the clearly negative side, LLM biases are currently unknown for research evaluation, and LLM scores are less transparent than citation counts. From a systemic perspective, a key issue is how introducing LLM-based indicators into research evaluation will change the behaviour of researchers. Whilst bibliometrics encourage some authors to target journals with high impact factors or to try to write highly cited work, LLM-based indicators may push them towards writing misleading abstracts and overselling their work in the hope of impressing the AI. Moreover, if AI-generated journal indicators replace impact factors, then this would encourage journals to allow authors to oversell their work in abstracts, threatening the integrity of the academic record.Keywords
Funding Information
- ESRC Metascience
This publication has 47 references indexed in Scilit:
- Evaluating the predictive capacity of ChatGPT for academic peer review outcomes across multiple platformsScientometrics, 2025
- Can ChatGPT be used to predict citation counts, readership, and social media interaction? An exploration among 2222 scientific abstractsScientometrics, 2024
- Factors associating with or predicting more cited or higher quality journal articles: An Annual Review of Information Science and Technology (ARIST) paperJournal of the Association for Information Science and Technology, 2023
- In which fields do higher impact journals publish higher quality articles?Scientometrics, 2023
- In which fields are citations indicators of research quality?Journal of the Association for Information Science and Technology, 2023
- When the Data Don’t Mean What They Say: Japan’s Comparative Underperformance in Citation Impact2020
- Self-correction of science: a comparative study of negative citations and post-publication peer reviewScientometrics, 2020
- Female citation impact superiority 1996–2018 in six out of seven English‐speaking nationsJournal of the Association for Information Science and Technology, 2019
- Co-existing Notions of Research Quality: A Framework to Study Context-specific Understandings of Good ResearchMinerva, 2019
- Factors affecting number of citations: a comprehensive review of the literatureScientometrics, 2016