T-Control: An Efficient Dynamic Tensor Rematerialization System for DNN Training
Open Access
- 22 March 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1949-1965
Abstract
With the continuous growth of model and batch sizes, DNN model training increasingly suffers from excessive memory consumption. Tensor rematerialization has emerged as an effective technique to enable training under limited memory constraints. However, dynamic rematerialization methods often underperform static approaches, primarily due to their greedy tensor eviction schedules and runtime overhead. In this work, we develop T-Control, a dynamic tensor rematerialization system for DNN model training. T-Control integrates the topology of the traced tensor dependency graph with real-time memory usage to make informed, adaptive tensor retention decisions, preserving critical tensors and reducing eviction-induced recomputation. Furthermore, by extending PyTorch's native memory manager with our fine-grained memory management strategy, T-control effectively improves memory utilization, thereby reducing unnecessary tensor eviction. Experimental results demonstrate that T-Control boosts throughput by up to 1.58× and 1.91× over state-of-the-art static and dynamic tensor rematerialization systems.Keywords
Funding Information
- National Natural Science Foundation of China (62032023, T2125013, and 62172391)
This publication has 41 references indexed in Scilit:
- OpenFold: retraining AlphaFold2 yields new insights into its learning mechanisms and capacity for generalizationNature Methods, 2024
- Fast and accurate variable batch size convolution neural network training on large scale distributed systemsConcurrency and Computation: Practice and Experience, 2022
- Social Media Sentiment Analysis Based on Dependency Graph and Co-occurrence GraphCognitive Computation, 2022
- A tensor framework for learning in structured domainsNeurocomputing, 2022
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Long Short-Term Memory2012
- On variants of shortest-path betweenness centrality and their generic computationSocial Networks, 2008
- Breadth-first heuristic searchArtificial Intelligence, 2006
- A faster algorithm for betweenness centrality*The Journal of Mathematical Sociology, 2001
- The Centrality Index of a GraphPsychometrika, 1966