Byte Latent Transformer: Patches Scale Better Than Tokens
Authors:
- Artidoro PagnoniAPArtidoro Pagnoni
- Ram PasunuruRPRam Pasunuru
- Pedro RodriguezPRPedro Rodriguez
- John NguyenJNJohn Nguyen
- Benjamin MullerBMBenjamin Muller
- Margaret LiMLMargaret Li
- Chunting ZhouCZChunting Zhou
- Lili YuLYLili Yu
- Jason WestonJWJason Weston
- Luke ZettlemoyerLZLuke Zettlemoyer
- Gargi GhoshGGGargi Ghosh
- Mike LewisMLMike Lewis
Preprint
Abstract
We introduce the Byte Latent Transformer (BLT), a new byte-level LLM architecture that, for the first time, matches tokenization-based LLM performance at scale with significant improvements in inference efficiency and robustness. BLT encodes bytes into dynamically sized patches, which serve as the primary units of computation. Patches are segmented based on the entropy of the next byte, allocating more compute and model capacity where increased data complexity demands it. We present the first FLOP controlled scaling study of byte-level models up to 8B parameters and 4T training bytes. Our results demonstrate the feasibility of scaling models trained on raw bytes without a fixed vocabulary. Both training and inference efficiency improve due to dynamically selecting long patches when data is predictable, along with qualitative improvements on reasoning and long tail generalization. Overall, for fixed inference costs, BLT shows significantly better scaling than tokenization-based models, by simultaneously growing both patch and model size.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- DataComp-LM: In search of the next generation of training sets for language models2024
- CUTE: Measuring LLMs’ Understanding of Their Tokens2024
- MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling2024
- PhonologyBench: Evaluating Phonological Skills of Large Language Models2024
- RoFormer: Enhanced transformer with Rotary Position EmbeddingNeurocomputing, 2023
- The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine TranslationTransactions of the Association for Computational Linguistics, 2022
- ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte ModelsTransactions of the Association for Computational Linguistics, 2022
- Canine: Pre-training an Efficient Tokenization-Free Encoder for Language RepresentationTransactions of the Association for Computational Linguistics, 2022
- HellaSwag: Can a Machine Really Finish Your Sentence?2019
- Learning to rank with (a lot of) word featuresInformation Retrieval Journal, 2009