The MorPhEMe Machine: An Addressable Neural Memory for Learning Knowledge-Regularized Deep Contextualized Chinese Embedding
- 9 February 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 32 (23299290) , 1673-1686
Abstract
Deep contextualized embeddings, as learned by large pre-training models, have proven highly effective in various downstream natural language processing tasks. However, the embedding space in these large models lacks explicit regularization, leading to underfitting and substantial costs during large-scale training on huge corpora. In this paper, we present a novel approach to learning deep contextualized embeddings, introducing linguistic knowledge regularization. Specifically, our proposed model, MorPhEMe (Morphology and Phonology Embedding Memory), features an external addressable memory with two additional addressable memories for storing morphology and phonology knowledge. MorPhEMe can be seamlessly stacked into a deep architecture. Notably different from existing pre-training models, MorPhEMe boasts two distinctive features: (1) compositional encoding and decompositional decoding facilitated by a dynamic addressing mechanism; and (2) explicit memory embedding regularization through cross-layer memory sharing. Theoretical analysis suggests that the inclusion of morphology and phonology enables MorPhEMe to reduce the modeling complexity of natural language sequences. We evaluate MorPhEMe across a diverse set of Chinese natural language processing tasks, including language modeling, word similarity computation, word analogy reasoning, relation extraction, and machine reading comprehension. Experimental results demonstrate that MorPhEMe, in contrast to state-of-the-art models, achieves remarkable improvements with fewer parameters and rapid convergence.Keywords
Funding Information
- Macao Young Scholars Program (AM201914)
- National Natural Science Foundation of China (62006041, 62076126, 61901221)
This publication has 35 references indexed in Scilit:
- MarkBERT: Marking Word Boundaries Improves Chinese BERT2023
- Pre-Training With Whole Word Masking for Chinese BERTIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- Morphology Matters: A Multilingual Language Modeling AnalysisTransactions of the Association for Computational Linguistics, 2021
- Efficient Content-Based Sparse Attention with Routing TransformersTransactions of the Association for Computational Linguistics, 2021
- A multi-feature fusion model for Chinese relation extraction with entity senseKnowledge-Based Systems, 2020
- BERT-Based Chinese Relation Extraction for Public SecurityIEEE Access, 2020
- Q8BERT: Quantized 8Bit BERT2019
- Deep Residual Learning for Image Recognition2016
- Morphological Analysis2015
- The Cambridge Handbook of Phonology2007