Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
Authors:
- Sabrina J. MielkeSMSabrina J. Mielke
- Zaid AlyafeaiZAZaid Alyafeai
- Elizabeth SaleskyESElizabeth Salesky
- Colin RaffelCRColin Raffel
- Manan DeyMDManan Dey
- Matthias GalléMGMatthias Gallé
- Arun RajaARArun Raja
- Chenglei SiCSChenglei Si
- Wilson Y. LeeWLWilson Y. Lee
- Benoît SagotBSBenoît Sagot
- Samson TanSTSamson Tan
Preprint