BioHanBERT: A Hanzi-aware Pre-trained Language Model for Chinese Biomedical Text Mining
Authors:
- 1 December 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1415-1420
Abstract
Unsupervised pre-trained language models (PLMs) have boosted the development of effective biomedical text mining models. But the biomedical texts contain a huge number of long-tail concepts and terminologies, which makes further pre-training on biomedical corpora relatively expensive (more biomedical corpora and more pre-training steps are needed). Nonetheless, this problem receives less attention in recent studies. In Chinese biomedical text, concepts and terminologies consist of Chinese characters, and Chinese characters are often composed of sub-character components which are also semantically informative; thus in order to enhance the semantics of biomedical concepts and terminologies, the use of a Chinese character’s component-level internal semantic information also appears to be reasonable.In this paper, we propose a novel hanzi-aware pre-trained language model for Chinese biomedical text mining, referred to as BioHanBERT (hanzi-aware BERT for Chinese biomedical text mining), utilizing the component-level internal semantic information of Chinese characters to enhance the semantics of Chinese biomedical concepts and terminologies, and thereby to reduce further pre-training costs. BioHanBERT first employs a Chinese character encoder to extract the component-level internal semantic feature of each Chinese character, and then fuse the character’s internal semantic feature and its contextual embedding extracted by BERT to enrich the representations of the concepts or terminologies containing the character. The results of extensive experiments show that our model is able to consistently outperform current state-of-the-art (SOTA) models in a wide range of Chinese biomedical natural language processing (NLP) tasks.Keywords
Funding Information
- Technology Development
- National Natural Science Foundation of China
This publication has 10 references indexed in Scilit:
- Improving Chinese Character Representation with Formation Graph Attention Network2021
- Learning Chinese Word Embeddings from Stroke, Structure and Pinyin of Characters2019
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019
- Deep Contextualized Word Representations2018
- Joint Embeddings of Chinese Words, Characters, and Fine-grained Subcharacter Components2017
- Learning Chinese Word Representations From Glyphs Of Characters2017
- Multi-Granularity Chinese Word Embedding2016
- Component-Enhanced Chinese Character Embeddings2015
- Radical-Enhanced Chinese Character Embedding2014
- Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction2011