Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimization
- 8 August 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
- Vol. 45 (3) , 1352-1365
Abstract
Transformer models have achieved state-of-the-art performance across a wide range of machine learning tasks. There is growing interest in training transformers on resource-constrained edge devices due to considerations, such as privacy, domain adaptation, and on-device scientific machine learning. However, the significant computational and memory demands required for transformer training often exceed the capabilities of an edge device. Leveraging low-rank tensor compression, this paper presents the first on-FPGA accelerator for transformer training. On the algorithm side, we present a bi-directional contraction flow for tensorized transformer training, significantly reducing the computational FLOPS and intralayer memory costs compared to existing tensor operations. On the hardware side, we store all highly compressed model parameters and gradient information on chip, creating an on-chip-memory-only framework for each stage in training. This reduces off-chip communication and minimizes latency and energy costs. Additionally, we implement custom computing kernels for each training stage and employ intralayer parallelism and pipe-lining to further enhance run-time and memory efficiency. Through experiments on transformer models within 36.7 to 93.5 MB using forward propagation (FP)-32 data formats on the ATIS dataset, our tensorized FPGA accelerator could conduct single-batch end-to-end training on the AMD Alevo U50 FPGA, with a memory budget of less than 6-MB BRAM and 22.5-MB URAM. Compared to uncompressed training on the NVIDIA RTX 3090 GPU, our on-FPGA training achieves a memory reduction of $30 \times $ to $51 \times $ . Our FPGA accelerator also achieves up to $4.0 \times $ less energy cost per epoch compared with tensor transformer training on an NVIDIA RTX 3090 GPU. As an initial result, this article highlights the significant potential of large-scale tensor training on edge devices.
Keywords
All Related Versions
Funding Information
- Intel Strategic Research Sectors (SRS)–Systems Integration SRS and Devices SRS
This publication has 37 references indexed in Scilit:
- Bebert: Efficient And Robust Binary Ensemble Bert2023
- FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Rounding2022
- Nonlinear tensor train format for deep neural network compressionNeural Networks, 2021
- Bayesian tensorized neural networks with automatic rank selectionNeurocomputing, 2021
- Compressing 3DCNNs based on tensor train decompositionNeural Networks, 2020
- Automatic Compiler Based FPGA Accelerator for CNN Training2019
- Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition2018
- SoK: Security and Privacy in Machine Learning2018
- Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks2018
- An FPGA-based processor for training convolutional neural networks2017