BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Top Cited Papers
Open Access
- 10 September 2019
- journal article
- research article
- Published by Oxford University Press (OUP) in Bioinformatics
- Vol. 36 (4) , 1234-1240
Abstract
Motivation: Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processing (NLP), extracting valuable information from biomedical literature has gained popularity among researchers, and deep learning has boosted the development of effective biomedical text mining models. However, directly applying the advancements in NLP to biomedical text mining often yields unsatisfactory results due to a word distribution shift from general domain corpora to biomedical corpora. In this article, we investigate how the recently introduced pre-trained language model BERT can be adapted for biomedical corpora. Results: We introduce BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), which is a domain-specific language representation model pre-trained on large-scale biomedical corpora. With almost the same architecture across tasks, BioBERT largely outperforms BERT and previous state-of-the-art models in a variety of biomedical text mining tasks when pre-trained on biomedical corpora. While BERT obtains performance comparable to that of previous state-of-the-art models, BioBERT significantly outperforms them on the following three representative biomedical text mining tasks: biomedical named entity recognition (0.62% F1 score improvement), biomedical relation extraction (2.80% F1 score improvement) and biomedical question answering (12.24% MRR improvement). Our analysis results show that pre-training BERT on biomedical corpora helps it to understand complex biomedical texts. Availability and implementation: We make the pre-trained weights of BioBERT freely available at https://github.com/naver/biobert-pretrained, and the source code for fine-tuning BioBERT available at https://github.com/dmis-lab/biobert.Keywords
All Related Versions
Funding Information
- National Research Foundation of Korea(NRF) funded by the Korea government (NRF-2017R1A2A1A17069645, NRF-2017M3C4A7065887, NRF-2014M3C9A3063541)
This publication has 22 references indexed in Scilit:
- An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competitionBMC Bioinformatics, 2015
- Extraction of relations between genes and diseases from text and large-scale data analysis: implications for translational researchBMC Bioinformatics, 2015
- The CHEMDNER corpus of chemicals and drugs and its annotation principlesJournal of Cheminformatics, 2015
- NCBI disease corpus: A resource for disease name recognition and concept normalizationJournal of Biomedical Informatics, 2014
- Glove: Global Vectors for Word Representation2014
- The SPECIES and ORGANISMS Resources for Fast and Accurate Identification of Taxonomic Names in TextPLOS ONE, 2013
- The EU-ADR corpus: Annotated drugs, diseases, targets, and their relationshipsJournal of Biomedical Informatics, 2012
- 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical textJournal of the American Medical Informatics Association, 2011
- LINNAEUS: A species name identification system for biomedical literatureBMC Bioinformatics, 2010
- Overview of BioCreative II gene mention recognitionGenome Biology, 2008