Learning Mutually Informed Representations for Characters and Subwords
Preprint
Abstract
Most pretrained language models rely on subword tokenization, which processes text as a sequence of subword tokens. However, different granularities of text, such as characters, subwords, and words, can contain different kinds of information. Previous studies have shown that incorporating multiple input granularities improves model generalization, yet very few of them outputs useful representations for each granularity. In this paper, we introduce the entanglement model, aiming to combine character and subword language models. Inspired by vision-language models, our model treats characters and subwords as separate modalities, and it generates mutually informed representations for both granularities as output. We evaluate our model on text classification, named entity recognition, POS-tagging, and character-level sequence labeling (intraword code-switching). Notably, the entanglement model outperforms its backbone language models, particularly in the presence of noisy texts and low-resource languages. Furthermore, the entanglement model even outperforms larger pre-trained models on all English sequence labeling tasks and classification tasks. We make our code publically available.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte ModelsTransactions of the Association for Computational Linguistics, 2022
- Canine: Pre-training an Efficient Tokenization-Free Encoder for Language RepresentationTransactions of the Association for Computational Linguistics, 2022
- MasakhaNER: Named Entity Recognition for African LanguagesTransactions of the Association for Computational Linguistics, 2021
- ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information2021
- CharBERT: Character-aware Pre-trained Language Model2020
- NutCracker at WNUT-2020 Task 2: Robustly Identifying Informative COVID-19 Tweets using Ensembling and Adversarial Training2020
- TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification2020
- BERTweet: A pre-trained language model for English Tweets2020
- Unsupervised Cross-lingual Representation Learning at Scale2020
- Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books2015