QuantuneV2: Compiler-Based Local Metric-Driven Mixed Precision Quantization for Practical Embedded AI Applications
Preprint
Abstract
Mixed-precision quantization methods have been proposed to reduce model size while minimizing accuracy degradation. However, existing studies require retraining and do not consider the computational overhead and intermediate representations (IR) generated during the compilation process, limiting their application at the compiler level. This computational overhead refers to the runtime latency caused by frequent quantization and dequantization operations during inference. Performing these operations at the individual operator level causes significant runtime delays. To address these issues, we propose QuantuneV2, a compiler-based mixed-precision quantization method designed for practical embedded AI applications. QuantuneV2 performs inference only twice, once before quantization and once after quantization, and operates with a computational complexity of O(n) that increases linearly with the number of model parameters. We also made the sensitivity analysis more stable by using local metrics like weights, activation values, the Signal to Quantization Noise Ratio, and the Mean Squared Error. We also cut down on computational overhead by choosing the best IR and using operator fusion. Experimental results show that QuantuneV2 achieved up to a 10.28 percent improvement in accuracy and a 12.52 percent increase in speed compared to existing methods across five models: ResNet18v1, ResNet50v1, SqueezeNetv1, VGGNet, and MobileNetv2. This demonstrates that QuantuneV2 enhances model performance while maintaining computational efficiency, making it suitable for deployment in embedded AI environments.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Deep learning inference optimisation for IoT: Conv2D-ReLU-BN layer fusion and quantisationThe Journal of Supercomputing, 2025
- Complexity-Aware Layer-Wise Mixed-Precision Schemes With SQNR-Based Fast AnalysisIEEE Access, 2023
- Quantune: Post-training quantization of convolutional neural networks using extreme gradient boosting for fast deploymentFuture Generation Computer Systems, 2022
- General Bitwidth Assignment for Efficient Deep Convolutional Neural Network QuantizationIEEE Transactions on Neural Networks and Learning Systems, 2021
- Mixed Precision Low-Bit Quantization of Neural Network Language Models for Speech RecognitionIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- Effective Loop Fusion in Polyhedral Compilation Using Fusion Conflict GraphsACM Transactions on Architecture and Code Optimization, 2020
- VecQ: Minimal Loss DNN Model Compression With Vectorized Weight QuantizationIEEE Transactions on Computers, 2020
- Low-bit Quantization of Neural Networks for Efficient Inference2019
- HAQ: Hardware-Aware Automated Quantization With Mixed Precision2019
- Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference2018