Reviewer Recommendations Using Document Vector Embeddings and a Publisher Database: Implementation and Evaluation
Open Access
- 15 February 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 21798-21811
Abstract
We develop and evaluate an automated data-driven framework for providing reviewer recommendations for submitted manuscripts. Given inputs comprising a set of manuscripts for review and a listing of a pool of prospective reviewers, our system uses a publisher database to extract papers authored by the reviewers from which a Paragraph Vector ( doc2vec ) neural network model is learned and used to obtain vector space embeddings of documents. Similarities between embeddings of an individual reviewer’s papers and a manuscript are then used to compute manuscript-reviewer match scores and to generate a ranked list of recommended reviewers for each manuscript. Our mainline proposed system uses full text versions of the reviewers’ papers, which we demonstrate performs significantly better than models developed based on abstracts alone, which has been the predominant paradigm in prior work. Direct retrieval of reviewer’s manuscripts from a publisher database reduces reviewer burden, ensures up-to-date data, and eliminates the potential for misuse through data manipulation. We also propose a useful evaluation methodology that addresses hyperparameter selection and enables indirect comparisons with alternative approaches and on prior datasets. Finally, the work also contributes a large scale retrospective reviewer matching dataset and evaluation that we hope will be useful for further research in this field. Our system is quite effective; for the mainline approach, expert judges rated 38% of the recommendations as Very Relevant ; 33% as Relevant ; 24% as Slightly Relevant ; and only 5% as Irrelevant .Keywords
All Related Versions
Funding Information
- U.S. National Science Foundation (CCF-1934962)
This publication has 13 references indexed in Scilit:
- BIOSSES: a semantic sentence similarity estimation system for the biomedical domainBioinformatics, 2017
- Computational support for academic peer reviewCommunications of the ACM, 2017
- The new automated IEEE INFOCOM review assignment systemIEEE Network, 2016
- Guest Editorial: Big Scholar Data Discovery and CollaborationIEEE Transactions on Big Data, 2016
- An Empirical Evaluation of doc2vec with Practical Insights into Document Embedding Generation2016
- Faked peer reviews prompt 64 retractionsNature, 2015
- Publishing: The peer-review scamNature, 2014
- Introduction to Information Retrieval2008
- Expertise modeling for matching papers with reviewers2007
- Mining for proposal reviewers2006