Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
Top Cited Papers
Open Access
- 31 January 2022
- journal article
- research article
- Published by MIT Press in Transactions of the Association for Computational Linguistics
- Vol. 10, 73-91
Abstract
Pipelined NLP systems have largely been superseded by end-to-end neural modeling, yet nearly all commonly used models still require an explicit tokenization step. While recent tokenization approaches based on data-derived subword lexicons are less brittle than manually engineered tokenizers, these techniques are not equally suited to all languages, and the use of any fixed vocabulary may limit a model’s ability to adapt. In this paper, we present Canine, a neural encoder that operates directly on character sequences—without explicit tokenization or vocabulary—and a pre-training strategy that operates either directly on characters or optionally uses subwords as a soft inductive bias. To use its finer-grained input effectively and efficiently, Canine combines downsampling, which reduces the input sequence length, with a deep transformer stack, which encodes context. Canine outperforms a comparable mBert model by 5.7 F1 on TyDi QA, a challenging multilingual benchmark, despite having fewer model parameters.Keywords
All Related Versions
This publication has 47 references indexed in Scilit:
- Character-level language modeling with hierarchical recurrent neural networks2017
- Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling2017
- Charagram: Embedding Words and Sentences via Character n-grams2016
- Neural Machine Translation of Rare Words with Subword Units2016
- Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models2016
- Multilingual Language Processing From Bytes2016
- Learning deep structured semantic models for web search using clickthrough data2013
- Bloom maps2008
- Introduction to the CoNLL-2003 shared task2003
- Introduction to the CoNLL-2002 shared task2002