Gradual Syntactic Label Replacement for Language Model Pre-Training
- 10 November 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 32 (23299290) , 486-496
Abstract
Pre-training serves as a foundation of recent NLP models, where language modeling tasks are performed over large texts. Typical models like BERT and GPT take the corpus as a whole and treat each word equally for language modeling. However, recent works show that the naturally existing frequency bias in the raw corpus may limit the power of the language model. In this paper, we propose a multi-stage training strategy that gradually increases the training vocabulary by modifying the training data. Specifically, we leverage the syntactic structure as a bridge for infrequent words and replace them with the corresponding syntactic labels, then we recover their original lexical surface for further training. Such strategy results in an easy-to-hard curriculum learning process, where the model learns the most common words and some basic syntax concepts, before recognizing a large number of uncommon words via their specific usages and the previously learned category knowledge. Experimental results show that such a method can improve the performance of both discriminative and generative pre-trained language models on benchmarks and various downstream tasks.Keywords
Funding Information
- National Natural Science Foundation of China (61976180, 61925601, 62276152)
- National Key R&D Program of China (2022ZD0160502)
This publication has 53 references indexed in Scilit:
- SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation2017
- ActiveCleanProceedings of the VLDB Endowment, 2016
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016
- Neural Machine Translation of Rare Words with Subword Units2016
- The LAMBADA dataset: Word prediction requiring a broad discourse context2016
- Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books2015
- Glove: Global Vectors for Word Representation2014
- Curriculum learning2009
- LabelMe: A Database and Web-Based Tool for Image AnnotationInternational Journal of Computer Vision, 2007
- Building a Large Annotated Corpus of English: The Penn Treebank1993