BERTMeSH: deep contextual representation learning for large-scale high-performance MeSH indexing with full text
Authors:
Open Access
- 25 September 2020
- journal article
- research article
- Published by Oxford University Press (OUP) in Bioinformatics
- Vol. 37 (5) , 684-692
Abstract
With the rapid increase of biomedical articles, large-scale automatic Medical Subject Headings (MeSH) indexing has become increasingly important. FullMeSH, the only method for large-scale MeSH indexing with full text, suffers from three major drawbacks: FullMeSH (i) uses Learning To Rank, which is time-consuming, (ii) can capture some pre-defined sections only in full text and (iii) ignores the whole MEDLINE database. We propose a computationally lighter, full text and deep-learning-based MeSH indexing method, BERTMeSH, which is flexible for section organization in full text. BERTMeSH has two technologies: (i) the state-of-the-art pre-trained deep contextual representation, Bidirectional Encoder Representations from Transformers (BERT), which makes BERTMeSH capture deep semantics of full text. (ii) A transfer learning strategy for using both full text in PubMed Central (PMC) and title and abstract (only and no full text) in MEDLINE, to take advantages of both. In our experiments, BERTMeSH was pre-trained with 3 million MEDLINE citations and trained on ∼1.5 million full texts in PMC. BERTMeSH outperformed various cutting-edge baselines. For example, for 20 K test articles of PMC, BERTMeSH achieved a Micro F-measure of 69.2%, which was 6.3% higher than FullMeSH with the difference being statistically significant. Also prediction of 20 K test articles needed 5 min by BERTMeSH, while it took more than 10 h by FullMeSH, proving the computational efficiency of BERTMeSH. Supplementary data are available at Bioinformatics onlineKeywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (61572139, 61872094)
- Shanghai Municipal Science and Technology Major Project (2017SHZDZX01)
- Information Technology Facility
- CAS-MPG Partner Institute for Computational Biology
- Shanghai Institute for Biological Sciences
- Chinese Academy of Sciences
- 111 Project (B18015)
- Shanghai Science & Technology (16JC1420402)
- Shanghai Municipal Science and Technology Major Project (2018SHZDZX01)
- ZJLab
- JST (JPMJAC1503)
- MEXT KAKENHI (19H04169)
- AIPSE by Academy of Finland
This publication has 18 references indexed in Scilit:
- DeepMeSH: deep semantic representation for improving large-scale MeSH indexingBioinformatics, 2016
- MeSHLabeler: improving the accuracy of large-scale MeSH indexing by integrating diverse evidenceBioinformatics, 2015
- An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competitionBMC Bioinformatics, 2015
- Threshold optimisation for multi-label classifiersPattern Recognition, 2013
- Enhanced clustering of biomedical documents using ensemble non-negative matrix factorizationInformation Sciences, 2011
- A Short Introduction to Learning to RankIEICE Transactions on Information and Systems, 2011
- Enhancing MEDLINE document clustering by incorporating MeSH semantic similarityBioinformatics, 2009
- Evaluation of query expansion using MeSH in PubMedInformation Retrieval Journal, 2008
- Exploring criteria for successful query expansion in the genomic domainInformation Retrieval Journal, 2008
- Long Short-Term MemoryNeural Computation, 1997