OAG-BERT: Pre-train Heterogeneous Entity-augmented Academic Language Models
Preprint
Abstract
To enrich language models with domain knowledge is crucial but difficult. Based on the world's largest public academic graph Open Academic Graph (OAG), we pre-train an academic language model, namely OAG-BERT, which integrates massive heterogeneous entities including paper, author, concept, venue, and affiliation. To better endow OAG-BERT with the ability to capture entity information, we develop novel pre-training strategies including heterogeneous entity type embedding, entity-aware 2D positional encoding, and span-aware entity masking. For zero-shot inference, we design a special decoding strategy to allow OAG-BERT to generate entity names from scratch. We evaluate the OAG-BERT on various downstream academic tasks, including NLP benchmarks, zero-shot entity inference, heterogeneous graph link prediction, and author name disambiguation. Results demonstrate the effectiveness of the proposed pre-training approach to both comprehending academic texts and modeling knowledge from heterogeneous entities. OAG-BERT has been deployed to multiple real-world applications, such as reviewer recommendations and paper tagging in the AMiner system. It is also available to the public through the CogDL package.Keywords
All Related Versions
This publication has 29 references indexed in Scilit:
- CONNA: Addressing Name Disambiguation on the FlyIEEE Transactions on Knowledge and Data Engineering, 2020
- POLAR++: Active One-Shot Personalized Article RecommendationIEEE Transactions on Knowledge and Data Engineering, 2019
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019
- A Scalable Hybrid Research Paper Recommender System for Microsoft Academic2019
- BioCreative V CDR task corpus: a resource for chemical disease relation extractionDatabase: The Journal of Biological Databases and Curation, 2016
- ChemProt-3.0: a global chemical biology diseases mappingDatabase: The Journal of Biological Databases and Curation, 2016
- An Overview of Microsoft Academic Service (MAS) and Applications2015
- NCBI disease corpus: A resource for disease name recognition and concept normalizationJournal of Biomedical Informatics, 2014
- GENIA corpus—a semantically annotated corpus for bio-textminingBioinformatics, 2003
- Word Reordering and a Dynamic Programming Beam Search Algorithm for Statistical Machine TranslationComputational Linguistics, 2003