TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation
Open Access
- 26 October 2026
- journal article
- research article
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 134 (11) , 495
Abstract
Pioneering token-based works such as Chameleon and Emu3 have established a foundation for multimodal unification but face challenges of high training computational overhead and limited comprehension performance due to a lack of high-level semantics. In this paper, we introduce TokLIP, a visual tokenizer that enhances comprehension by semanticizing vector-quantized (VQ) tokens and incorporating CLIP-level semantics while enabling end-to-end multimodal autoregressive training with standard VQ tokens. TokLIP integrates a low-level discrete VQ tokenizer with a ViT-based token encoder to capture high-level continuous semantics. Unlike previous approaches (e.g., VILA-U) that discretize high-level features, TokLIP disentangles training objectives for comprehension and generation, allowing the direct application of advanced VQ tokenizers without the need for tailored quantization operations. Our empirical results demonstrate that TokLIP achieves exceptional data efficiency, empowering visual tokens with high-level semantic understanding while enhancing low-level generative capacity, making it well-suited for autoregressive Transformers in both open-world comprehension and generation tasks. The code and models are available at https://github.com/TencentARC/TokLIP .Keywords
Funding Information
- City University of Hong Kong
This publication has 52 references indexed in Scilit:
- TokenPacker: Efficient Visual Projector for Multimodal LLMInternational Journal of Computer Vision, 2025
- Animal-CLIP: A Dual-Prompt Enhanced Vision-Language Model for Animal Action RecognitionInternational Journal of Computer Vision, 2025
- Effectively Leveraging CLIP for Generating Situational Summaries of Images and VideosInternational Journal of Computer Vision, 2025
- VideoQA in the Era of LLMs: An Empirical StudyInternational Journal of Computer Vision, 2025
- MMBench: Is Your Multi-modal Model an All-Around Player?2024
- Source-Free Domain Adaptation Guided by Vision and Vision-Language Pre-trainingInternational Journal of Computer Vision, 2024
- LLMFormer: Large Language Model for Open-Vocabulary Semantic SegmentationInternational Journal of Computer Vision, 2024
- CLIP-Adapter: Better Vision-Language Models with Feature AdaptersInternational Journal of Computer Vision, 2023
- A Diagram is Worth a Dozen Images2016
- Food-101 – Mining Discriminative Components with Random Forests2014