OAG-BERT: Towards a Unified Backbone Language Model for Academic Knowledge Services
Authors:
Open Access
- 14 August 2022
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 3418-3428
Abstract
Academic Knowledge Services have substantially facilitated the development of human science and technology, providing a plenitude of useful research tools. However, many applications highly depend on ad-hoc models and expensive human labeling to understand professional contents, hindering deployments in real world. To create a unified backbone language model for various knowledge-intensive academic knowledge mining challenges, based on the world's largest public academic graph Open Academic Graph (OAG), we pre-train an academic language model, namely OAG-BERT, to integrate massive heterogeneous entity knowledge beyond scientific corpora. We develop novel pre-training strategies along with zero-shot inference techniques. OAG-BERT's superior performance on 9 knowledge-intensive academic tasks (including 2 demo applications) demonstrates its qualification to serve as a foundation for academic knowledge services. Its zero-shot capability also offers great potential to mitigate the need of costly annotations. OAG-BERT has been deployed to multiple real-world applications, such as reviewer recommendations for NSFC (National Nature Science Foundation of China) and paper tagging in the AMiner system. All codes and pre-trained models are available via the CogDL.Keywords
All Related Versions
Funding Information
- NSFC (61836013; 61825602)
This publication has 21 references indexed in Scilit:
- Construction of the Literature Graph in Semantic Scholar2018
- metapath2vec2017
- Recommendation of scholarly venues based on dynamic user interestsJournal of Informetrics, 2017
- An Overview of Microsoft Academic Service (MAS) and Applications2015
- Venue Recommendation: Submitting Your Paper with Style2012
- Scholarly paper recommendation via user's recent research interests2010
- ArnetMiner2008
- Scientific paper summarization using citation summary networks2008
- HT06, tagging paper, taxonomy, Flickr, academic article, to read2006
- Word Reordering and a Dynamic Programming Beam Search Algorithm for Statistical Machine TranslationComputational Linguistics, 2003