A Model for Every User and Budget: Label-Free and Personalized Mixed-Precision Quantization
Preprint
Abstract
Recent advancement in Automatic Speech Recognition (ASR) has produced large AI models, which become impractical for deployment in mobile devices. Model quantization is effective to produce compressed general-purpose models, however such models may only be deployed to a restricted sub-domain of interest. We show that ASR models can be personalized during quantization while relying on just a small set of unlabelled samples from the target domain. To this end, we propose myQASR, a mixed-precision quantization method that generates tailored quantization schemes for diverse users under any memory requirement with no fine-tuning. myQASR automatically evaluates the quantization sensitivity of network layers by analysing the full-precision activation values. We are then able to generate a personalised mixed-precision quantization scheme for any pre-determined memory budget. Results for large-scale ASR models show how myQASR improves performance for specific genders, languages, and speakers.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- PTQ4ViT: Post-training Quantization for Vision Transformers with Twin Uniform Quantization2022
- Low-Bit Quantization of Transformer for Audio Speech Recognition2022
- Disentangled Loss for Low-Bit Quantization-Aware Training2022
- Integer-Only Zero-Shot Quantization for Efficient Speech Recognition2022
- Pruning and quantization for deep neural network acceleration: A surveyNeurocomputing, 2021
- Diversifying Sample Generation for Accurate Data-Free Quantization2021
- Combining Weight Pruning and Knowledge Distillation For CNN Compression2021
- ZeroQ: A Novel Zero Shot Quantization Framework2020
- Low-bit Quantization Needs Good Distribution2020
- Librispeech: An ASR corpus based on public domain audio books2015