BiQGEMM: Matrix Multiplication with Lookup Table for Binary-Coding-Based Quantized DNNs
Authors:
- 1 November 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-14
Abstract
The number of parameters in deep neural networks (DNNs) is rapidly increasing to support complicated tasks and to improve model accuracy. Correspondingly, the amount of computations and required memory footprint increase as well. Quantization is an efficient method to address such concerns by compressing DNNs such that computations can be simplified while required storage footprint is significantly reduced. Unfortunately, commercial CPUs and GPUs do not fully support quantization because only fixed data transfers (such as 32 bits) are allowed. As a result, even if weights are quantized (by a non-uniform quantization scheme) into a few bits, CPUs and GPUs may not access multiple quantized weights without memory bandwidth waste. Success of quantization in practice, hence, relies on an efficient computation engine design, especially for matrix multiplication that is a basic computation engine in most DNNs. In this paper, we propose a novel matrix multiplication method, called BiQGEMM, dedicated to quantized DNNs. BiQGEMM can access multiple quantized weights simultaneously in one instruction. In addition, BiQGEMM pre-computes intermediate results that are highly redundant when quantization leads to limited available computation space. Since pre-computed values are stored in lookup tables and reused, BiQGEMM achieves lower amount of overall computations. Our extensive experimental results show that BiQGEMM presents higher performance than conventional schemes when DNNs are quantized.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- A Survey to Predict the Trend of AI-able Server Evolution in the CloudIEEE Access, 2018
- Self-Attention with Relative Position Representations2018
- Network Sketching: Exploiting Binary Structure in Deep CNNs2017
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016
- Intel Math Kernel Library2014
- Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets2013
- Benchmarking GPUs to tune dense linear algebra2008
- Long Short-Term MemoryNeural Computation, 1997
- First draft of a report on the EDVACIEEE Annals of the History of Computing, 1993
- Learning representations by back-propagating errorsNature, 1986