TNT: Improving Chunkwise Training for Test-Time Memorization
Preprint
Abstract
Recurrent neural networks (RNNs) with deep test-time memorization modules, such as Titans and TTT, represent a promising, linearly-scaling paradigm distinct from Transformers. While these expressive models do not yet match the peak performance of state-of-the-art Transformers, their potential has been largely untapped due to prohibitively slow training and low hardware utilization. Existing parallelization methods force a fundamental conflict governed by the chunksize hyperparameter: large chunks boost speed but degrade performance, necessitating a fixed, suboptimal compromise. To solve this challenge, we introduce TNT, a novel training paradigm that decouples training efficiency from inference performance through a two-stage process. Stage one is an efficiency-focused pre-training phase utilizing a hierarchical memory. A global module processes large, hardware-friendly chunks for long-range context, while multiple parallel local modules handle fine-grained details. Crucially, by periodically resetting local memory states, we break sequential dependencies to enable massive context parallelization. Stage two is a brief fine-tuning phase where only the local memory modules are adapted to a smaller, high-resolution chunksize, maximizing accuracy with minimal overhead. Evaluated on Titans and TTT models, TNT achieves a substantial acceleration in training speed-up to 17 times faster than the most accurate baseline configuration - while simultaneously improving model accuracy. This improvement removes a critical scalability barrier, establishing a practical foundation for developing expressive RNNs and facilitating future work to close the performance gap with Transformers.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Exponential Capacity of Dense Associative MemoriesPhysical Review Letters, 2024
- Parallelizing Linear Transformers with the Delta Rule over Sequence Length2024
- Gated Slot Attention for Efficient Linear-Time Sequence Modeling2024
- Towards Scalable and Stable Parallelization of Nonlinear RNNs2024
- RWKV: Reinventing RNNs for the Transformer Era2023
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness2022
- ABC: Attention with Bounded-memory Control2022
- Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent NetworksNeural Computation, 1992