Bojan Denić

This paper proposes an SQNR-based method for the design of bfloat16-inspired 16-bit floating-point representations for ReLU activations. The method exploits the analogy between floating-point formats and piecewise-uniform quantizers and uses SQNR as a performance metric to evaluate alternative bit allocations between the significand and exponent fields for a given input probability density function (PDF). For ReLU activations, the PDF is derived from a Laplacian pre-activation model, showing that ReLU outputs are non-negative and that the sign bit in bfloat16 is redundant. The redundant bit is therefore reallocated within the fixed 16-bit budget, yielding two representations: FP16^I, with the bit added to the significand, and FP16^II, with the bit added to the exponent. Using the ReLU PDF, closed-form SQNR models are obtained for both representations and used to characterize their performance relative to bfloat16. An analysis over a wide range of input variances shows that FP16^I provides a 6.02 dB SQNR improvement over bfloat16, whereas FP16^II maintains the bfloat16 SQNR level within a substantially wider variance range. These SQNR models are further applied to guide the selection between FP16^I and FP16^II for CNN training based on the observed variance range of pre-ReLU activations. It is demonstrated that replacing the bfloat16 ReLU representation with the selected alternative in a bfloat16-quantized CNN results in training performance closer to that of the FP32-based configuration.

The 32-bit floating-point (FP32) format has many useful applications, particularly in computing and neural network systems. The classic 32-bit fixed-point (FXP32) format often introduces lower quality of representation (i.e., precision), making it unsuitable for real deployment, despite offering faster computations and reduced computational cost, which positively impacts energy efficiency. In this paper, we propose a switched FXP32 format able to compete with or surpass the widely used FP32 format across a wide variance range. It actually proposes switching between the possible values of key parameters according to the variance level of the data modeled with the Laplacian distribution. Precision analysis is achieved using the signal-to-quantization noise ratio (SQNR) as a performance metric, introduced based on the analogy between digital formats and quantization. Theoretical SQNR results provided in a wide range of variance confirm the design objectives. Experimental and simulation results obtained using neural network weights further support the approach. The strong agreement between the experiment, simulation, and theory indicates the efficiency of this proposal in encoding Laplacian data, as well as its potential applicability in neural networks.

This paper introduces a novel approach that bridges the floating-point (FP) format, widely utilized in diverse fields for data representation, with the μ-law companding quantizer, proposing a method for designing and linearizing the μ-law companding quantizer to yield a piecewise uniform quantizer tailored to the FP format. A key outcome of the paper is a closed-form approximate expression for closely and efficiently evaluating the FP format’s performance for data with the Laplacian distribution. This expression offers generality across various bit rates and data variances, markedly reducing the computational complexity of FP performance evaluation compared to prior methods reliant on summation of a large number of terms. By facilitating the evaluation of FP format performance, this research substantially aids in the selection of the optimal bit rates, crucial for digital representation quality, dynamic range, computational overhead, and energy efficiency. The numerical calculations spanning a wide range of data variances provided for some commonly used FP versions with an 8-bit exponent have demonstrated that the proposed closed-form expression closely approximates FP format performance.

The research conducted in this paper targets to examine the efficiency of the 24-bits floating-point (FP24) format on data modelled by the Gaussian distribution. To accomplish this goal, we exploit a connection that exists between binary formats and quantization, so we apply the objective measures such as distortion and SQNR (signal-to-quantization noise ratio) to estimate the performance of the FP24 format. In particular, performance investigation of this format is performed in the theoretical domain for both mean-squared error (MSE) and absolute error (AE) measures. Results for both metrics are provided in the wide dynamic range of the input data variances and show that the SQNR is high (103.79 dB for MSE and 105.19 dB for AE) and varies negligibly. Therefore, the FP24 format can be considered as an excellent candidate for representing Gaussian data in practical variance-sensitive applications.

The main issue when dealing with the non-adaptive scalar quantizers is their sensitivity to variance-mismatch, the effect that occurs when the data variance differs from the one used for the quantizer design. In this paper, we consider the influence of that effect in low-rate (2-bit) uniform scalar quantization (USQ) of Laplacian source and also we propose adequate measure to suppress it. Particularly, the approach we propose represents the upgraded version of the previous approaches used to improve performance of the single quantizer. It is based on dual-mode quantization that combines two 2-bit USQs (with adequately chosen parameters) to process input data, selected by applying the special rule. Analysis conducted in theoretical domain has shown that the proposed approach is less sensitive to variance-mismatch, making the dual-mode USQ more efficient in terms of robustness than the single USQ. Also, a gain is achieved compared to other 2-bit quantizer solutions. Experimental results are also provided for quantization of weights of the multi-layer perceptron (MLP) neural network, where good matching with the theoretical results is observed. Due to these achievements, we believe that the solution we propose can be a good choice for compression of non-stationary data modeled by Laplacian distribution, such as neural network parameters.

This paper considers the design of a binary scalar quantizer of Laplacian source and its application in compressed neural networks. The quantizer performance is investigated in a wide dynamic range of data variances, and for that purpose, we derive novel closed-form expressions. Moreover, we propose two selection criteria for the variance range of interest. Binary quantizers are further implemented for compressing neural network weights and its performance is analysed for a simple classification task. Good matching between theory and experiment is observed and a great possibility for implementation is indicated.

Achieving real-time inference is one of the major issues in contemporary neural network applications, as complex algorithms are frequently being deployed to mobile devices that have constrained storage and computing power. Moving from a full-precision neural network model to a lower representation by applying quantization techniques is a popular approach to facilitate this issue. Here, we analyze in detail and design a 2-bit uniform quantization model for Laplacian source due to its significance in terms of implementation simplicity, which further leads to a shorter processing time and faster inference. The results show that it is possible to achieve high classification accuracy (more than 96% in the case of MLP and more than 98% in the case of CNN) by implementing the proposed model, which is competitive to the performance of the other quantization solutions with almost optimal precision.

A novel 2‐bit adaptive delta modulation (ADM) algorithm is presented based on uniform scalar quantization and fractional linear prediction (FLP) for encoding the signals modelled by a Gaussian probability density function. The study focusses on two major areas: realization of a 2‐bit adaptive quantizer based on Q ‐function approximation that significantly facilitates quantizer design; and implementation of a recently introduced FLP approach with the memory of two samples, which replaces the first‐order linear prediction used in standard ADM algorithms and enables improved performance without increasing transmission costs. It furthermore represents the first implementation of FLP in signal encoding, therefore confirming its applicability in a real signal‐processing scenario. Based on the performance analysis conducted on a real speech signal, the proposed ADM algorithm with FLP is demonstrated to outperform other 2‐bit ADM baselines by a large margin for the gain in signal‐to‐noise ratio achieved over a wide dynamic range of input signals. The results of this research indicate that ADM with adaptive quantization based on Q‐function approximation and adaptive FLP represents a promising solution for encoding/compression of correlated time‐varying signals following the Gaussian distribution.

A compression method based on non-uniform binary scalar quantization, designed for the memoryless Laplacian source with zero-mean and unit variance, is analyzed in this paper. Two quantizer design approaches are presented that investigate the effect of clipping with the aim of reducing the quantization noise, where the minimal mean-squared error distortion is used to determine the optimal clipping factor. A detailed comparison of both models is provided, and the performance evaluation in a wide dynamic range of input data variances is also performed. The observed binary scalar quantization models are applied in standard signal processing tasks, such as speech and image quantization, but also to quantization of neural network parameters. The motivation behind the binary quantization of neural network weights is the model compression by a factor of 32, which is crucial for implementation in mobile or embedded devices with limited memory and processing power. The experimental results follow well the theoretical models, confirming their applicability in real-world applications.