Beyond Peak Performance: Comparing the Real Performance of AI-Optimized FPGAs and GPUs
Authors:
- 1 December 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 10-19
Abstract
The growing importance and compute demands of artificial intelligence (AI) have led to the emergence of domain-optimized hardware platforms. For example, Nvidia GPUs introduced specialized tensor cores for matrix operations to speed up deep learning (DL) computation, resulting in very high peak throughput up to 130 int8 TOPS in the T4 GPU. Recently, Intel introduced its first AI-optimized 14nm FPGA, the Stratix 10 NX, with in-fabric AI tensor blocks that offer estimated peak performance up to 143 int8 TOPS, comparable to 12nm GPUs. However, what matters in practice is not the peak performance but the actual achievable performance on target workloads. This depends mainly on the utilization of the tensor units, and the system-level overheads to send data to/from the accelerator. This paper presents the first performance evaluation of Intel's AI-optimized FPGA, the Stratix 10 NX, in comparison to the latest accessible AI-optimized GPUs, the Nvidia T4 and V100, on a large suite of real-time DL inference workloads. We enhance a re-implementation of the Brainwave NPU overlay architecture to utilize the FPGA's AI tensor blocks, and develop toolchain support that allows users to program tensor blocks purely through software, without FPGA EDA tools in the loop. We first compare the Stratix 10 NX NPU against Stratix 10 GX/MX versions with no tensor blocks, and then present detailed core compute and system-level performance comparisons to the T4 and V100 GPUs. We show that our enhanced NPU on Stratix 10 NX achieves better tensor block utilization than GPUs, resulting in 24× and 12× average compute speedups over the T4 and V100 GPUs at batch-6. Even with relaxed latency constraints that allow a batch size of 32, we still achieve average speedups of 5× and 2× against T4 and V100 GPUs, respectively. On a system-level, the FPGA's fine-grained flexibility with its integrated 100 Gbps Ethernet allows for remote access at 10× and 2× less system overhead latency than local access to a V100 GPU via 128 Gbps PCIe for short and long sequence RNNs, respectively.Keywords
This publication has 34 references indexed in Scilit:
- Mastering the game of Go without human knowledgeNature, 2017
- In-Datacenter Performance Analysis of a Tensor Processing Unit2017
- ESE2017
- Can FPGAs Beat GPUs in Accelerating Next-Generation Deep Neural Networks?2017
- FPGA-based accelerator for long short-term memory recurrent neural networks2017
- Accelerating recurrent neural networks in analytics servers: Comparison of FPGA, CPU, GPU, and ASIC2016
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Reducing GPU offload latency via fine-grained CPU-GPU synchronization2013
- The long sentence: A disservice to science in the Internet ageBioEssays, 2011
- Long Short-Term MemoryNeural Computation, 1997