A 7-nm Four-Core Mixed-Precision AI Chip With 26.2-TFLOPS Hybrid-FP8 Training, 104.9-TOPS INT4 Inference, and Workload-Aware Throttling
- 13 November 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Solid-State Circuits
- Vol. 57 (1) , 182-197
Abstract
Reduced precision computation is a key enabling factor for energy-efficient acceleration of deep learning (DL) applications. This article presents a 7-nm four-core mixed-precision artificial intelligence (AI) chip that supports four compute precisions—FP16, Hybrid-FP8 (HFP8), INT4, and INT2—to support diverse application demands for training and inference. The chip leverages cutting-edge algorithmic advances to demonstrate leading-edge power efficiency for 8-bit floating-point (FP8) training and INT4 inference without model accuracy degradation. A new HFP8 format combined with separation of the floating- and fixed-point pipelines and aggressive circuit/architecture optimization enables performance improvements while maintaining high compute utilization. A high-bandwidth ring protocol enables efficient data communication, while power management using workload-aware clock throttling maximizes performance within a given power budget. The AI chip demonstrates 3.58-TFLOPS/W peak energy efficiency and 26.2-TFLOPS peak performance for HFP8 iso-accuracy training, and 16.9-TOPS/W peak energy efficiency and 104.9-TOPS peak performance for INT4 iso-accuracy inference.Keywords
This publication has 19 references indexed in Scilit:
- A 0.11 pJ/Op, 0.32-128 TOPS, Scalable Multi-Chip-Module-based Deep Neural Network Accelerator with Ground-Reference Signaling in 16nm2019
- A 16-nm Always-On DNN Processor With Adaptive Clocking and Multi-Cycle Banked SRAMsIEEE Journal of Solid-State Circuits, 2019
- 7.7 LNPU: A 25.3TFLOPS/W Sparse Deep-Neural-Network Learning Processor with Fine-Grained Mixed Precision of FP8-FP162019
- DNN Engine: A 28-nm Timing-Error Tolerant Sparse Deep Neural Network Processor for IoT ApplicationsIEEE Journal of Solid-State Circuits, 2018
- A Scalable Multi- TeraOPS Deep Learning Processor Core for AI Trainina and Inference2018
- In-Datacenter Performance Analysis of a Tensor Processing Unit2017
- 14.5 Envision: A 0.26-to-10TOPS/W subword-parallel dynamic-voltage-accuracy-frequency-scalable Convolutional Neural Network processor in 28nm FDSOI2017
- XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks2016
- Deep Residual Learning for Image Recognition2016
- 14.5 Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks2016