Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models
Top Cited Papers
Open Access
- 1 January 2023
- conference paper
- Published by Association for Computational Linguistics (ACL)
- p. 9904-9923
Abstract
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, Yulia Tsvetkov. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine TranslationTransactions of the Association for Computational Linguistics, 2022
- Quality at a Glance: An Audit of Web-Crawled Multilingual DatasetsTransactions of the Association for Computational Linguistics, 2022
- Gender Bias and Under-Representation in Natural Language Processing Across Human Languages2021
- Byte Pair Encoding is Suboptimal for Language Model Pretraining2020
- Improving Multilingual Models with Language-Clustered Vocabularies2020
- Unsupervised Cross-lingual Representation Learning at Scale2020
- BPE-Dropout: Simple and Effective Subword Regularization2020
- Toward Gender-Inclusive Coreference Resolution2020
- Social Biases in NLP Models as Barriers for Persons with Disabilities2020
- Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot TranslationTransactions of the Association for Computational Linguistics, 2017