Rethinking Output Alignment For 1-bit Post-Training Quantization of Large Language Models
Preprint
Abstract
Large Language Models (LLMs) deliver strong performance across a wide range of NLP tasks, but their massive sizes hinder deployment on resource-constrained devices. To reduce their computational and memory burden, various compression techniques have been proposed, including quantization, pruning, and knowledge distillation. Among these, post-training quantization (PTQ) is widely adopted for its efficiency, as it requires no retraining and only a small dataset for calibration, enabling low-cost deployment. Recent advances for post-training quantization have demonstrated that even sub-4-bit methods can maintain most of the original model performance. However, 1-bit quantization that converts floating-point weights to \(\pm\)1, remains particularly challenging, as existing 1-bit PTQ methods often suffer from significant performance degradation compared to the full-precision models. Specifically, most of existing 1-bit PTQ approaches focus on weight alignment, aligning the full-precision model weights with those of the quantized models, rather than directly aligning their outputs. Although the output-matching approach objective is more intuitive and aligns with the quantization goal, naively applying it in 1-bit LLMs often leads to notable performance degradation. In this paper, we investigate why and under what conditions output-matching fails, in the context of 1-bit LLM quantization. Based on our findings, we propose a novel data-aware PTQ approach for 1-bit LLMs that explicitly accounts for activation error accumulation while keeping optimization efficient. Empirical experiments demonstrate that our solution consistently outperforms existing 1-bit PTQ methods with minimal overhead.
Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and AccelerationGetMobile: Mobile Computing and Communications, 2025
- ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers2022
- PIQA: Reasoning about Physical Commonsense in Natural LanguageProceedings of the AAAI Conference on Artificial Intelligence, 2020
- BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions2019
- HellaSwag: Can a Machine Really Finish Your Sentence?2019
- Bi-Real Net: Enhancing the Performance of 1-Bit CNNs with Improved Representational Capability and Advanced Training Algorithm2018
- Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering2018