Chengtao Lv
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Chengtao Lv has a total of 33 co-authors. Here are the top co-authors by number of shared publications:
- Xianglong Liu
Beihang University, Beijing, China
6 shared publications - Haotong Qin
ETH Zurich
4 shared publications Jinyang Guo
Beihang University
3 shared publicationsDacheng Tao
The University of Sydney
2 shared publicationsHong Chen
Beihang University
2 shared publications- Hong Chen
School of Computer Science and Engineering, Beihang University, Beijing, China
2 shared publications
Latest Publications
- 15 July 2025
- Published by Elsevier in Neural Networks
- Vol. 192, 107856
A survey of low-bit large language models: Basics, systems, and algorithms
- 15 July 2025
- Published by Elsevier in Neural Networks
- Vol. 192, 107856
A survey of low-bit large language models: Basics, systems, and algorithms
- Top Cited PapersOpen Access
- 30 December 2024
- Published by Springer Nature in Visual Intelligence
- Vol. 2 (1) , 1-13
An empirical study of LLaMA3 quantization: from LLMs to MLLMs
The LLaMA family, a collection of foundation language models ranging from 7B to 65B parameters, has become one of the most powerful open-source large language models (LLMs) and the popular LLM backbone of multi-modal large language models (MLLMs), widely used in computer vision and natural language understanding tasks. In particular, LLaMA3 models have recently been released and have achieved impressive performance in various domains with super-large scale pre-training on over 15T tokens of data. Given the wide application of low-bit quantization for LLMs in resource-constrained scenarios, we explore LLaMA3’s capabilities when quantized to low bit-width. This exploration can potentially provide new insights and challenges for the low-bit quantization of LLaMA3 and other future LLMs, especially in addressing performance degradation issues that suffer in LLM compression. Specifically, we comprehensively evaluate the 10 existing post-training quantization and LoRA fine-tuning (LoRA-FT) methods of LLaMA3 on 1-8 bits and various datasets to reveal the low-bit quantization performance of LLaMA3. To uncover the capabilities of low-bit quantized MLLM, we assessed the performance of the LLaMA3-based LLaVA-Next-8B model under 2-4 ultra-low bits with post-training quantization methods. Our experimental results indicate that LLaMA3 still suffers from non-negligible degradation in linguistic and visual contexts, particularly under ultra-low bit widths. This highlights the significant performance gap at low bit-width that needs to be addressed in future developments. We expect that this empirical study will prove valuable in advancing future models, driving LLMs and MLLMs to achieve higher accuracy at lower bit to enhance practicality.
- 30 December 2024
- Published by Springer Nature in Visual Intelligence
- Vol. 2 (1) , 1-13
Top Cited PapersOpen AccessAn empirical study of LLaMA3 quantization: from LLMs to MLLMs
The LLaMA family, a collection of foundation language models ranging from 7B to 65B parameters, has become one of the most powerful open-source large language models (LLMs) and the popular LLM backbone of multi-modal large language models (MLLMs), widely used in computer vision and natural language understanding tasks. In particular, LLaMA3 models have recently been released and have achieved impressive performance in various domains with super-large scale pre-training on over 15T tokens of data. Given the wide application of low-bit quantization for LLMs in resource-constrained scenarios, we explore LLaMA3’s capabilities when quantized to low bit-width. This exploration can potentially provide new insights and challenges for the low-bit quantization of LLaMA3 and other future LLMs, especially in addressing performance degradation issues that suffer in LLM compression. Specifically, we comprehensively evaluate the 10 existing post-training quantization and LoRA fine-tuning (LoRA-FT) methods of LLaMA3 on 1-8 bits and various datasets to reveal the low-bit quantization performance of LLaMA3. To uncover the capabilities of low-bit quantized MLLM, we assessed the performance of the LLaMA3-based LLaVA-Next-8B model under 2-4 ultra-low bits with post-training quantization methods. Our experimental results indicate that LLaMA3 still suffers from non-negligible degradation in linguistic and visual contexts, particularly under ultra-low bit widths. This highlights the significant performance gap at low bit-width that needs to be addressed in future developments. We expect that this empirical study will prove valuable in advancing future models, driving LLMs and MLLMs to achieve higher accuracy at lower bit to enhance practicality.
- Open Access
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 10572-10581
QVD: Post-training Quantization for Video Diffusion Models
Recently, video diffusion models (VDMs) have garnered significant attention due to their notable advancements in generating coherent and realistic video content. However, processing multiple frame features concurrently, coupled with the considerable model size, results in high latency and extensive memory consumption, hindering their broader application. Post-training quantization (PTQ) is an effective technique to reduce memory footprint and improve computational efficiency. Unlike image diffusion, we observe that the temporal features, which are integrated into all frame features, exhibit pronounced skewness. Furthermore, we investigate significant inter-channel disparities and asymmetries in the activation of video diffusion models, resulting in low coverage of quantization levels by individual channels and increasing the challenge of quantization. To address these issues, we introduce the first PTQ strategy tailored for video diffusion models, dubbed QVD. Specifically, we propose the High Temporal Discriminability Quantization (HTDQ) method, designed for temporal features, which retains the high discriminability of quantized features, providing precise temporal guidance for all video frames. In addition, we present the Scattered Channel Range Integration (SCRI) method which aims to improve the coverage of quantization levels across individual channels. Experimental validations across various models, datasets, and bit-width settings demonstrate the effectiveness of our QVD in terms of diverse metrics. In particular, we achieve near-lossless performance degradation on W8A8, outperforming the current methods by 205.12 in FVD.
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 10572-10581
Open AccessQVD: Post-training Quantization for Video Diffusion Models
Recently, video diffusion models (VDMs) have garnered significant attention due to their notable advancements in generating coherent and realistic video content. However, processing multiple frame features concurrently, coupled with the considerable model size, results in high latency and extensive memory consumption, hindering their broader application. Post-training quantization (PTQ) is an effective technique to reduce memory footprint and improve computational efficiency. Unlike image diffusion, we observe that the temporal features, which are integrated into all frame features, exhibit pronounced skewness. Furthermore, we investigate significant inter-channel disparities and asymmetries in the activation of video diffusion models, resulting in low coverage of quantization levels by individual channels and increasing the challenge of quantization. To address these issues, we introduce the first PTQ strategy tailored for video diffusion models, dubbed QVD. Specifically, we propose the High Temporal Discriminability Quantization (HTDQ) method, designed for temporal features, which retains the high discriminability of quantized features, providing precise temporal guidance for all video frames. In addition, we present the Scattered Channel Range Integration (SCRI) method which aims to improve the coverage of quantization levels across individual channels. Experimental validations across various models, datasets, and bit-width settings demonstrate the effectiveness of our QVD in terms of diverse metrics. In particular, we achieve near-lossless performance degradation on W8A8, outperforming the current methods by 205.12 in FVD.
- Open Access
DB-LLM: Accurate Dual-Binarization for Efficient LLMs
Large language models (LLMs) have significantly advanced the field of natural language processing, while the expensive memory and computation consumption impede their practical deployment. Quantization emerges as one of the most effective methods for improving the computational efficiency of LLMs. However, existing ultra-low-bit quantization always causes severe accuracy drops. In this paper, we empirically relieve the micro and macro characteristics of ultra-low bit quantization and present a novel Dual-Binarization method for LLMs, namely DB-LLM. For the micro-level, we take both the accuracy advantage of 2-bit-width and the efficiency advantage of binarization into account, introducing Flexible Dual Binarization (FDB). By splitting 2-bit quantized weights into two independent sets of binaries, FDB ensures the accuracy of representations and introduces flexibility, utilizing the efficient bitwise operations of binarization while retaining the inherent high sparsity of ultra-low bit quantization. For the macro-level, we find the distortion that exists in the prediction of LLM after quantization, which is specified as the deviations related to the ambiguity of samples. We propose the Deviation-Aware Distillation (DAD) method, enabling the model to focus differently on various samples. Comprehensive experiments show that our DB-LLM not only significantly surpasses the current State-of-The-Art (SoTA) in ultra-low bit quantization (eg, perplexity decreased from 9.64 to 7.23), but also achieves an additional 20\% reduction in computational consumption compared to the SOTA method under the same bit-width. Our code will be released soon.
Open AccessDB-LLM: Accurate Dual-Binarization for Efficient LLMs
Large language models (LLMs) have significantly advanced the field of natural language processing, while the expensive memory and computation consumption impede their practical deployment. Quantization emerges as one of the most effective methods for improving the computational efficiency of LLMs. However, existing ultra-low-bit quantization always causes severe accuracy drops. In this paper, we empirically relieve the micro and macro characteristics of ultra-low bit quantization and present a novel Dual-Binarization method for LLMs, namely DB-LLM. For the micro-level, we take both the accuracy advantage of 2-bit-width and the efficiency advantage of binarization into account, introducing Flexible Dual Binarization (FDB). By splitting 2-bit quantized weights into two independent sets of binaries, FDB ensures the accuracy of representations and introduces flexibility, utilizing the efficient bitwise operations of binarization while retaining the inherent high sparsity of ultra-low bit quantization. For the macro-level, we find the distortion that exists in the prediction of LLM after quantization, which is specified as the deviations related to the ambiguity of samples. We propose the Deviation-Aware Distillation (DAD) method, enabling the model to focus differently on various samples. Comprehensive experiments show that our DB-LLM not only significantly surpasses the current State-of-The-Art (SoTA) in ultra-low bit quantization (eg, perplexity decreased from 9.64 to 7.23), but also achieves an additional 20\% reduction in computational consumption compared to the SOTA method under the same bit-width. Our code will be released soon.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 8719-8730
DB-LLM: Accurate Dual-Binarization for Efficient LLMs
Hong Chen, Chengtao Lv, Liang Ding, Haotong Qin, Xiabin Zhou, Yifu Ding, Xuebo Liu, Min Zhang, Jinyang Guo, Xianglong Liu, Dacheng Tao. Findings of the Association for Computational Linguistics ACL 2024. 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 8719-8730
Open AccessDB-LLM: Accurate Dual-Binarization for Efficient LLMs
Hong Chen, Chengtao Lv, Liang Ding, Haotong Qin, Xiabin Zhou, Yifu Ding, Xuebo Liu, Min Zhang, Jinyang Guo, Xianglong Liu, Dacheng Tao. Findings of the Association for Computational Linguistics ACL 2024. 2024.
- Open Access
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 63-67
CDMNet: Contrastive Distribution Mapped Network for Infrared Small Target Detection
Single-frame infrared small target (SIRST) detection is an extremely challenging task due to its low signal-to-noise ratio and low contrast. Previous methods fail to achieve promising performance as they do not consider the analogous and blurred background surrounding. To this end, we first propose a prototype-based contrastive loss (PCL) by modeling the foreground targets and the surrounding nearest backgrounds. As a result, the prototypes of different categories in the latent space could be far away, which enables the model to make clear decisions on the boundaries of infrared small targets. Moreover, previous methods neglect the distribution inconsistency caused by feature fusion in U-shaped architecture. Therefore, we design a multi-scale distribution-mapped fusion (MDMF) module, which greatly mitigates the distribution inconsistency issue. We incorporate the proposed PCL and MDMF module into the existing SIRST detection method to construct a new SIRST detection framework called Contrastive Distribution Mapped Network (CDMNet). Extensive experiments on two infrared small target datasets, NUDT-SIRST and IRSTD-1k, demonstrate that our model outperforms current competitive models on a variety of metrics.
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 63-67
Open AccessCDMNet: Contrastive Distribution Mapped Network for Infrared Small Target Detection
Single-frame infrared small target (SIRST) detection is an extremely challenging task due to its low signal-to-noise ratio and low contrast. Previous methods fail to achieve promising performance as they do not consider the analogous and blurred background surrounding. To this end, we first propose a prototype-based contrastive loss (PCL) by modeling the foreground targets and the surrounding nearest backgrounds. As a result, the prototypes of different categories in the latent space could be far away, which enables the model to make clear decisions on the boundaries of infrared small targets. Moreover, previous methods neglect the distribution inconsistency caused by feature fusion in U-shaped architecture. Therefore, we design a multi-scale distribution-mapped fusion (MDMF) module, which greatly mitigates the distribution inconsistency issue. We incorporate the proposed PCL and MDMF module into the existing SIRST detection method to construct a new SIRST detection framework called Contrastive Distribution Mapped Network (CDMNet). Extensive experiments on two infrared small target datasets, NUDT-SIRST and IRSTD-1k, demonstrate that our model outperforms current competitive models on a variety of metrics.