Chinese Clinical Named Entity Recognition with ALBERT and MHA Mechanism
Open Access
- 23 May 2022
- journal article
- research article
- Published by Wiley in Evidence-Based Complementary and Alternative Medicine
- Vol. 2022, 1-9
Abstract
Traditional clinical named entity recognition methods fail to balance the effectiveness of feature extraction of unstructured text and the complexity of neural network models. We propose a model based on ALBERT and a multihead attention (MHA) mechanism to solve this problem. Structurally, the model first obtains character-level word embeddings through the ALBERT pretraining language model, then inputs the word embeddings into the iterated dilated convolutional neural network model to quickly extract global semantic information, and decodes the predicted labels through conditional random fields to obtain the optimal label sequence. Also, we apply the MHA mechanism to capture intercharacter dependencies from multiple aspects. Furthermore, we use the RAdam optimizer to boost the convergence speed and improve the generalization ability of our model. Experimental results show that our model achieves an F1 score of 85.63% on the CCKS-2019 dataset—an increase of 4.36% compared to the baseline model.Keywords
Funding Information
- Central Public Welfare Research Institutes (ZZ140319-W, 61772078)
- National Natural Science Foundation of China (ZZ140319-W, 61772078)
This publication has 11 references indexed in Scilit:
- Chinese clinical named entity recognition with variant neural structures based on BERT methodsJournal of Biomedical Informatics, 2020
- Biomedical Named Entity Recognition Based on Self‐supervised Deep Belief NetworkChinese Journal of Electronics, 2020
- DTranNER: biomedical named entity recognition with deep learning-based label-label transition modelBMC Bioinformatics, 2020
- Extracting comprehensive clinical information for breast cancer using deep learning methodsInternational Journal of Medical Informatics, 2019
- Incorporating dictionaries into deep neural networks for the Chinese clinical named entity recognitionJournal of Biomedical Informatics, 2019
- Patient Knowledge Distillation for BERT Model Compression2019
- Marginal Likelihood Training of BiLSTM-CRF for Biomedical Named Entity Recognition from Disjoint Label Sets2018
- GRAM-CNN: a deep learning approach with local context for named entity recognition in biomedical textBioinformatics, 2017
- Anatomical Entity Recognition with a Hierarchical Framework Augmented by External ResourcesPLOS ONE, 2014
- Hybrid methods for improving information access in clinical documents: concept, assertion, and relation identificationJournal of the American Medical Informatics Association, 2011