Haokun Lin
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Haokun Lin has a total of 69 co-authors. Here are the top co-authors by number of shared publications:
- Zhenan Sun
Institute of Automation, Chinese Academy of Sciences, Beijing, China
7 shared publications - Yichen Wu
Harvard University
6 shared publications - Lianwei Yang
Institute of Automation, Chinese Academy of Sciences
3 shared publications - Qingyi Gu
Institute of Automation, Chinese Academy of Sciences, Beijing, China
3 shared publications - Caifeng Shan
Nanjing University
2 shared publications - Chen Ma
City University of Hong Kong
2 shared publications
Latest Publications
- 1 November 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 36 (11) , 17093-17108
LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generation
- 1 November 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 36 (11) , 17093-17108
LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generation
- Open Access
- 26 October 2026
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 134 (11) , 495
TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation
Pioneering token-based works such as Chameleon and Emu3 have established a foundation for multimodal unification but face challenges of high training computational overhead and limited comprehension performance due to a lack of high-level semantics. In this paper, we introduce TokLIP, a visual tokenizer that enhances comprehension by semanticizing vector-quantized (VQ) tokens and incorporating CLIP-level semantics while enabling end-to-end multimodal autoregressive training with standard VQ tokens. TokLIP integrates a low-level discrete VQ tokenizer with a ViT-based token encoder to capture high-level continuous semantics. Unlike previous approaches (e.g., VILA-U) that discretize high-level features, TokLIP disentangles training objectives for comprehension and generation, allowing the direct application of advanced VQ tokenizers without the need for tailored quantization operations. Our empirical results demonstrate that TokLIP achieves exceptional data efficiency, empowering visual tokens with high-level semantic understanding while enhancing low-level generative capacity, making it well-suited for autoregressive Transformers in both open-world comprehension and generation tasks. The code and models are available at https://github.com/TencentARC/TokLIP .
- 26 October 2026
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 134 (11) , 495
Open AccessTokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation
Pioneering token-based works such as Chameleon and Emu3 have established a foundation for multimodal unification but face challenges of high training computational overhead and limited comprehension performance due to a lack of high-level semantics. In this paper, we introduce TokLIP, a visual tokenizer that enhances comprehension by semanticizing vector-quantized (VQ) tokens and incorporating CLIP-level semantics while enabling end-to-end multimodal autoregressive training with standard VQ tokens. TokLIP integrates a low-level discrete VQ tokenizer with a ViT-based token encoder to capture high-level continuous semantics. Unlike previous approaches (e.g., VILA-U) that discretize high-level features, TokLIP disentangles training objectives for comprehension and generation, allowing the direct application of advanced VQ tokenizers without the need for tailored quantization operations. Our empirical results demonstrate that TokLIP achieves exceptional data efficiency, empowering visual tokens with high-level semantic understanding while enhancing low-level generative capacity, making it well-suited for autoregressive Transformers in both open-world comprehension and generation tasks. The code and models are available at https://github.com/TencentARC/TokLIP .
- 5 October 2026
- Published by Springer Nature
- p. 348-366
Before Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoning
- 5 October 2026
- Published by Springer Nature
- p. 348-366
Before Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoning
- Open Access
- 25 August 2026
- Published by Springer Nature in Machine Intelligence Research
- Vol. 23 (5) , 1145-1161
Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs
Recent advances in diffusion large language models (dLLMs) have introduced a promising alternative to autoregressive (AR) LLMs for natural language generation tasks, leveraging full attention and denoising-based decoding strategies. However, the deployment of these models on edge devices remains challenging due to their massive parameter scale and high resource demands. While post-training quantization (PTQ) has emerged as a widely adopted technique for compressing AR LLMs, its applicability to dLLMs remains largely unexplored. In this work, we present the first systematic study on quantizing diffusion-based language models. We begin by identifying the presence of activation outliers, characterized by abnormally large activation values that dominate the dynamic range. These outliers pose a key challenge to low-bit quantization, as they make it difficult to preserve precision for the majority of values. More importantly, we implement state-of-the-art PTQ methods and conduct a comprehensive evaluation across multiple task types and model variants. Our analysis is structured along four key dimensions: bit-width, quantization method, task category, and model type. Through this multi-perspective evaluation, we offer that practical insights into the quantization behavior of dLLMs under different configurations. We hope that our findings provide a foundation for future research in efficient dLLM deployment. Our code is publicly available at https://github.com/MessiX77/QDLM .
- 25 August 2026
- Published by Springer Nature in Machine Intelligence Research
- Vol. 23 (5) , 1145-1161
Open AccessQuantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs
Recent advances in diffusion large language models (dLLMs) have introduced a promising alternative to autoregressive (AR) LLMs for natural language generation tasks, leveraging full attention and denoising-based decoding strategies. However, the deployment of these models on edge devices remains challenging due to their massive parameter scale and high resource demands. While post-training quantization (PTQ) has emerged as a widely adopted technique for compressing AR LLMs, its applicability to dLLMs remains largely unexplored. In this work, we present the first systematic study on quantizing diffusion-based language models. We begin by identifying the presence of activation outliers, characterized by abnormally large activation values that dominate the dynamic range. These outliers pose a key challenge to low-bit quantization, as they make it difficult to preserve precision for the majority of values. More importantly, we implement state-of-the-art PTQ methods and conduct a comprehensive evaluation across multiple task types and model variants. Our analysis is structured along four key dimensions: bit-width, quantization method, task category, and model type. Through this multi-perspective evaluation, we offer that practical insights into the quantization behavior of dLLMs under different configurations. We hope that our findings provide a foundation for future research in efficient dLLM deployment. Our code is publicly available at https://github.com/MessiX77/QDLM .
- Open Access
- 15 June 2026
- Published by Association for Computing Machinery (ACM)
- p. 2371-2380
DapQ-DiT: Distribution-Aware Post-Training Quantization for Efficient Generative Tasks in Diffusion Transformers
- 15 June 2026
- Published by Association for Computing Machinery (ACM)
- p. 2371-2380
Open AccessDapQ-DiT: Distribution-Aware Post-Training Quantization for Efficient Generative Tasks in Diffusion Transformers
- 10 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. PP (10518215) , 1
Singular Value Fine-tuning for Few-Shot Class-Incremental Learning
Class-Incremental Learning (CIL) aims to prevent catastrophic forgetting of previously learned classes while sequentially incorporating new ones. Few-shot CIL (FSCIL) presents a more challenging setting by introducing novel classes with only a few labeled examples, which significantly increases the risk of overfitting beyond the standard challenges faced in CIL. While catastrophic forgetting has been extensively studied, overfitting in FSCIL, especially when using large foundation models, has received comparatively limited attention. To fill this gap, we propose Singular Value Fine-tuning for FSCIL (SVFCL) and compare it with existing Parameter Efficient Fine-Tuning (PEFT) methods like prompt tuning and Low-Rank Adaptation (LoRA). Specifically, SVFCL applies singular value decomposition to the foundation model weights. For each task, the singular vectors are kept fixed while only the singular values are fine-tuned, and the updated singular values are subsequently merged to obtain the adapted model. This simple yet effective approach not only alleviates the problem of catastrophic forgetting but also mitigates overfitting more effectively by significantly reducing the number of trainable parameters. The effectiveness of SVFCL is validated through extensive experiments on four benchmark datasets, accompanied by thorough ablation studies and in-depth discussions.
- 10 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. PP (10518215) , 1
Singular Value Fine-tuning for Few-Shot Class-Incremental Learning
Class-Incremental Learning (CIL) aims to prevent catastrophic forgetting of previously learned classes while sequentially incorporating new ones. Few-shot CIL (FSCIL) presents a more challenging setting by introducing novel classes with only a few labeled examples, which significantly increases the risk of overfitting beyond the standard challenges faced in CIL. While catastrophic forgetting has been extensively studied, overfitting in FSCIL, especially when using large foundation models, has received comparatively limited attention. To fill this gap, we propose Singular Value Fine-tuning for FSCIL (SVFCL) and compare it with existing Parameter Efficient Fine-Tuning (PEFT) methods like prompt tuning and Low-Rank Adaptation (LoRA). Specifically, SVFCL applies singular value decomposition to the foundation model weights. For each task, the singular vectors are kept fixed while only the singular values are fine-tuned, and the updated singular values are subsequently merged to obtain the adapted model. This simple yet effective approach not only alleviates the problem of catastrophic forgetting but also mitigates overfitting more effectively by significantly reducing the number of trainable parameters. The effectiveness of SVFCL is validated through extensive experiments on four benchmark datasets, accompanied by thorough ablation studies and in-depth discussions.
- Open Access
- 23 January 2026
- Published by Wiley
Efficient Diffusion Language Models: A Comprehensive Survey
Diffusion language models (dLLMs) have recently emerged as a promising alternative to autoregressive (AR) language models, offering competitive performance with a fundamentally different generation paradigm. Instead of producing tokens strictly left-to-right, dLLMs iteratively refine partially masked sequences, enabling bidirectional context modeling and supporting parallel token updates. This design creates new opportunities for accelerating generation and improving controllability. However, practical deployment of dLLMs remains challenging due to the high cost of training at scale and the lack of mature inference optimizations such as cache-friendly decoding and robust parallel sampling. In this work, we provide a comprehensive overview of recent progress on efficient dLLMs. To reflect how efficiency bottlenecks arise across the full lifecycle of dLLMs, spanning training, decoding, and deployment, we organize existing approaches into five categories: Training, Inference, Context, Framework, and Multimodality. For each category, we summarize representative methods and highlight their motivations, key techniques, and empirical trade-offs. Finally, we discuss open challenges and outline promising directions for future research toward high-throughput, robust, and practically deployable diffusion language models. Github Repository: https://github.com/FelixMessi/Awesome-Efficient-dLLMs .
- 23 January 2026
- Published by Wiley
Open AccessEfficient Diffusion Language Models: A Comprehensive Survey
Diffusion language models (dLLMs) have recently emerged as a promising alternative to autoregressive (AR) language models, offering competitive performance with a fundamentally different generation paradigm. Instead of producing tokens strictly left-to-right, dLLMs iteratively refine partially masked sequences, enabling bidirectional context modeling and supporting parallel token updates. This design creates new opportunities for accelerating generation and improving controllability. However, practical deployment of dLLMs remains challenging due to the high cost of training at scale and the lack of mature inference optimizations such as cache-friendly decoding and robust parallel sampling. In this work, we provide a comprehensive overview of recent progress on efficient dLLMs. To reflect how efficiency bottlenecks arise across the full lifecycle of dLLMs, spanning training, decoding, and deployment, we organize existing approaches into five categories: Training, Inference, Context, Framework, and Multimodality. For each category, we summarize representative methods and highlight their motivations, key techniques, and empirical trade-offs. Finally, we discuss open challenges and outline promising directions for future research toward high-throughput, robust, and practically deployable diffusion language models. Github Repository: https://github.com/FelixMessi/Awesome-Efficient-dLLMs .
- 18 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 7510-7521
Scale Up Composed Image Retrieval Learning via Modification Text Generation
Composed Image Retrieval (CIR) aims to search an image of interest using a combination of a reference image and modification text as the query. Despite recent advancements, this task remains challenging due to limited training data and laborious triplet annotation processes. To address this issue, this paper proposes to synthesize the training triplets to augment the training resource for the CIR problem. Specifically, we commence by training a modification text generator exploiting large-scale multimodal models and scale up the CIR learning throughout both the pretraining and fine-tuning stages. During pretraining, we leverage the trained generator to directly create Modification Text-oriented Synthetic Triplets (MTST) conditioned on pairs of images. For fine-tuning, we first synthesize reverse modification text to connect the target image back to the reference image. Subsequently, we devise a two-hop alignment strategy to incrementally close the semantic gap between the multimodal pair and the target image. We initially learn an implicit prototype utilizing both the original triplet and its reversed version in a cycle manner, followed by combining the implicit prototype feature with the modification text to facilitate accurate alignment with the target image. Extensive experiments validate the efficacy of the generated triplets and confirm that our proposed methodology attains competitive recall on both the CIRR and FashionIQ benchmarks. Codes and datasets will be made publicly accessible.
- 18 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 7510-7521
Scale Up Composed Image Retrieval Learning via Modification Text Generation
Composed Image Retrieval (CIR) aims to search an image of interest using a combination of a reference image and modification text as the query. Despite recent advancements, this task remains challenging due to limited training data and laborious triplet annotation processes. To address this issue, this paper proposes to synthesize the training triplets to augment the training resource for the CIR problem. Specifically, we commence by training a modification text generator exploiting large-scale multimodal models and scale up the CIR learning throughout both the pretraining and fine-tuning stages. During pretraining, we leverage the trained generator to directly create Modification Text-oriented Synthetic Triplets (MTST) conditioned on pairs of images. For fine-tuning, we first synthesize reverse modification text to connect the target image back to the reference image. Subsequently, we devise a two-hop alignment strategy to incrementally close the semantic gap between the multimodal pair and the target image. We initially learn an implicit prototype utilizing both the original triplet and its reversed version in a cycle manner, followed by combining the implicit prototype feature with the modification text to facilitate accurate alignment with the target image. Extensive experiments validate the efficacy of the generated triplets and confirm that our proposed methodology attains competitive recall on both the CIRR and FashionIQ benchmarks. Codes and datasets will be made publicly accessible.
- Open Access
Singular Value Fine-tuning for Few-Shot Class-Incremental Learning
Class-Incremental Learning (CIL) aims to prevent catastrophic forgetting of previously learned classes while sequentially incorporating new ones. The more challenging Few-shot CIL (FSCIL) setting further complicates this by providing only a limited number of samples for each new class, increasing the risk of overfitting in addition to standard CIL challenges. While catastrophic forgetting has been extensively studied, overfitting in FSCIL, especially with large foundation models, has received less attention. To fill this gap, we propose the Singular Value Fine-tuning for FSCIL (SVFCL) and compared it with existing approaches for adapting foundation models to FSCIL, which primarily build on Parameter Efficient Fine-Tuning (PEFT) methods like prompt tuning and Low-Rank Adaptation (LoRA). Specifically, SVFCL applies singular value decomposition to the foundation model weights, keeping the singular vectors fixed while fine-tuning the singular values for each task, and then merging them. This simple yet effective approach not only alleviates the forgetting problem but also mitigates overfitting more effectively while significantly reducing trainable parameters. Extensive experiments on four benchmark datasets, along with visualizations and ablation studies, validate the effectiveness of SVFCL. The code will be made available.
Open AccessSingular Value Fine-tuning for Few-Shot Class-Incremental Learning
Class-Incremental Learning (CIL) aims to prevent catastrophic forgetting of previously learned classes while sequentially incorporating new ones. The more challenging Few-shot CIL (FSCIL) setting further complicates this by providing only a limited number of samples for each new class, increasing the risk of overfitting in addition to standard CIL challenges. While catastrophic forgetting has been extensively studied, overfitting in FSCIL, especially with large foundation models, has received less attention. To fill this gap, we propose the Singular Value Fine-tuning for FSCIL (SVFCL) and compared it with existing approaches for adapting foundation models to FSCIL, which primarily build on Parameter Efficient Fine-Tuning (PEFT) methods like prompt tuning and Low-Rank Adaptation (LoRA). Specifically, SVFCL applies singular value decomposition to the foundation model weights, keeping the singular vectors fixed while fine-tuning the singular values for each task, and then merging them. This simple yet effective approach not only alleviates the forgetting problem but also mitigates overfitting more effectively while significantly reducing trainable parameters. Extensive experiments on four benchmark datasets, along with visualizations and ablation studies, validate the effectiveness of SVFCL. The code will be made available.