CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
Preprint
Abstract
Pipelined NLP systems have largely been superseded by end-to-end neural modeling, yet nearly all commonly-used models still require an explicit tokenization step. While recent tokenization approaches based on data-derived subword lexicons are less brittle than manually engineered tokenizers, these techniques are not equally suited to all languages, and the use of any fixed vocabulary may limit a model's ability to adapt. In this paper, we present CANINE, a neural encoder that operates directly on character sequences, without explicit tokenization or vocabulary, and a pre-training strategy that operates either directly on characters or optionally uses subwords as a soft inductive bias. To use its finer-grained input effectively and efficiently, CANINE combines downsampling, which reduces the input sequence length, with a deep transformer stack, which encodes context. CANINE outperforms a comparable mBERT model by 2.8 F1 on TyDi QA, a challenging multilingual benchmark, despite having 28% fewer model parameters.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse LanguagesTransactions of the Association for Computational Linguistics, 2020
- Character-level Representations Improve DRS-based Semantic Parsing Even in the Age of BERT2020
- Extending Multilingual BERT to Low-Resource Languages2020
- MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer2020
- Character-Level Language Modeling with Deeper Self-AttentionProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language ModelProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Neural Lattice Language ModelsTransactions of the Association for Computational Linguistics, 2018
- Language Modeling for Morphologically Rich Languages: Character-Aware Modeling for Word-Level PredictionTransactions of the Association for Computational Linguistics, 2018
- Enriching Word Vectors with Subword InformationTransactions of the Association for Computational Linguistics, 2017
- Fully Character-Level Neural Machine Translation without Explicit SegmentationTransactions of the Association for Computational Linguistics, 2017