AI-assisted peer review
Top Cited Papers
Open Access
- 25 January 2021
- journal article
- review article
- Published by Springer Nature in Humanities and Social Sciences Communications
- Vol. 8 (1) , 1-11
Abstract
The scientific literature peer review workflow is under strain because of the constant growth of submission volume. One response to this is to make initial screening of submissions less time intensive. Reducing screening and review time would save millions of working hours and potentially boost academic productivity. Many platforms have already started to use automated screening tools, to prevent plagiarism and failure to respect format requirements. Some tools even attempt to flag the quality of a study or summarise its content, to reduce reviewers’ load. The recent advances in artificial intelligence (AI) create the potential for (semi) automated peer review systems, where potentially low-quality or controversial studies could be flagged, and reviewer-document matching could be performed in an automated manner. However, there are ethical concerns, which arise from such approaches, particularly associated with bias and the extent to which AI systems may replicate bias. Our main goal in this study is to discuss the potential, pitfalls, and uncertainties of the use of AI to approximate or assist human decisions in the quality assurance and peer-review process associated with research outputs. We design an AI tool and train it with 3300 papers from three conferences, together with their reviews evaluations. We then test the ability of the AI in predicting the review score of a new, unobserved manuscript, only using its textual content. We show that such techniques can reveal correlations between the decision process and other quality proxy measures, uncovering potential biases of the review process. Finally, we discuss the opportunities, but also the potential unintended consequences of these techniques in terms of algorithmic bias and ethical concerns.Keywords
This publication has 34 references indexed in Scilit:
- Defining and Characterizing Open Peer Review: A Review of the LiteratureJournal of Scholarly Publishing, 2013
- Beyond the paperNature, 2013
- Bias in peer reviewJournal of the American Society for Information Science and Technology, 2012
- Peer review in a changing world: An international study measuring the attitudes of researchersJournal of the American Society for Information Science and Technology, 2012
- Unblinking eyes: the ethics of automating surveillanceEthics and Information Technology, 2012
- Computer-Supported Peer Review in a Law School ContextSSRN Electronic Journal, 2012
- Scientific peer reviewAnnual Review of Information Science and Technology, 2011
- Peering Inside Peer Review with Bayesian ModelsLecture Notes in Computer Science, 2011
- Attention web designers: You have 50 milliseconds to make a good first impression!Behaviour & Information Technology, 2006
- Manuscript evaluation by journal referees and editors: Randomness or bias?Behavioral and Brain Sciences, 1982