FxpNet: Training a deep convolutional neural network in fixed-point representation
Authors:
- 1 May 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 15, 2494-2501
Abstract
We introduce FxpNet, a framework to train deep convolutional neural networks with low bit-width arithmetics in both forward pass and backward pass. During training FxpNet further reduces the bit-width of stored parameters (also known as primal parameters) by adaptively updating their fixed-point formats. These primal parameters are usually represented in the full resolution of floating-point values in previous binarized and quantized neural networks. In FxpNet, during forward pass fixed-point primal weights and activations are first binarized before computation, while in backward pass all gradients are represented as low resolution fixed-point values and then accumulated to corresponding fixed-point primal parameters. To have highly efficient implementations in FPGAs, ASICs and other dedicated devices, FxpNet introduces Integer Batch Normalization (IBN) and Fixed-point ADAM (FxpADAM) methods to further reduce the required floating-point operations, which will save considerable power and chip area. The evaluation on CIFAR-10 dataset indicates the effectiveness that FxpNet with 12-bit primal parameters and 12-bit gradients achieves comparable prediction accuracy with state-of-the-art binarized and quantized neural networks.Keywords
This publication has 10 references indexed in Scilit:
- Going Deeper with Embedded FPGA Platform for Convolutional Neural Network2016
- Efficient and accurate approximations of nonlinear convolutional networks2015
- Going deeper with convolutions2015
- DaDianNao: A Machine-Learning Supercomputer2014
- Fast and Robust Neural Network Joint Models for Statistical Machine Translation2014
- Deep convolutional neural networks for LVCSR2013
- NeuFlow: Dataflow vision processing system-on-a-chip2012
- NeuFlow: A runtime reconfigurable dataflow processor for vision2011
- A highly scalable Restricted Boltzmann Machine FPGA implementation2009
- Back propagation simulations using limited precision calculations2002