An FPGA-Based Efficient Streaming Vector Processing Engine for Transformer-Based Models
Authors:
- 10 May 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 722-727
Abstract
Transformer-based models have obtained extensive success. As their linear operations have been significantly accelerated by a wide range of approaches, nonlinear operations tend to have limited hardware efficiency and become the performance bottleneck. Prior works to accelerate nonlinear operations suffer from either low area efficiency with poor instruction set architecture (ISA) or limited throughput due to the loop-carried dependency in reduce operations. In this paper, we propose a vector processing engine (VPE) with new streaming ISA to achieve flexible streaming execution for nonlinear operations and obtain better hardware efficiency. Moreover, we relax the loop-carried dependency in reduce operations with look-ahead dataflow optimization and improve the throughput of nonlinear operations. Experimental results on Xilinx U200 FPGA show that VPE can outperform other FPGA vector processing units on throughput by 1.14 x -3 x for softmax, layer normalization, GELU across different vector sizes.Keywords
This publication has 8 references indexed in Scilit:
- FET-OPU: A Flexible and Efficient FPGA-Based Overlay Processor for Transformer Networks2023
- HPTA: A High Performance Transformer Accelerator Based on FPGA2023
- DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation2022
- RISC-VTF: RISC-V Based Extended Instruction Set for Transformer2021
- NPE: An FPGA-based Overlay Processor for Natural Language Processing2021
- Hardware Acceleration of Fully Quantized BERT for Efficient Natural Language Processing2021
- FTRANS2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019