BinaryRelax: A Relaxation Approach for Training Deep Neural Networks with Quantized Weights

Authors:
  • PYPenghang Yin
    Penghang Yin
  • SZShuai Zhang
    Shuai Zhang
  • JLJiancheng Lyu
    Jiancheng Lyu
  • SOStanley Osher
    Stanley Osher
  • YQYingyong Qi
    Yingyong Qi
Publisher website
Google Scholar
Abstract
We propose BinaryRelax, a simple two-phase algorithm, for training deep neural networks with quantized weights. The set constraint that characterizes the quantization of weights is not imposed until the late stage of training, and a sequence of pseudo quantized weights is maintained. Specifically, we relax the hard constraint into a continuous regularizer via a Moreau envelope, which turns out to be the squared Euclidean distance to the set of quantized weights. The pseudo quantized weights are obtained by linearly interpolating between the float weights and their quantizations. A continuation strategy is adopted to push the weights toward the quantized state by gradually increasing the regularization parameter. In the second phase, an exact quantization scheme with a small learning rate is invoked to guarantee fully quantized weights. We test BinaryRelax on the benchmark CIFAR and ImageNet color image datasets to demonstrate the superiority of the relaxed quantization approach and the improved accuracy o...
Funding Information
  • Air Force Office of Scientific Research (FA 9550-15-0073)
  • Office of Naval Research (N00014-16-1-2157)
  • National Science Foundation (DMS-1522383, IIS-1632935)
  • U.S. Department of Energy (DE-SC00183838)

This publication has 18 references indexed in Scilit: