Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding
Authors:
Top Cited Papers
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 13700-13710
Abstract
Large language models have demonstrated impressive universal capabilities across a wide range of open-ended tasks and have extended their utility to encompass multi-modal conversations. However, existing methods encounter challenges in effectively handling both image and video understanding, particularly with limited visual tokens. In this work, we introduce Chat-UniVi, a Unified Vision-language model capable of comprehending and engaging in conver-sations involving images and videos through a unified visual representation. Specifically, we employ a set of dynamic visual tokens to uniformly represent images and videos. This representation framework empowers the model to ef-ficiently utilize a limited number of visual tokens to simul-taneously capture the spatial details necessary for images and the comprehensive temporal relationship required for videos. Moreover, we leverage a multi-scale representation, enabling the model to perceive both high-level seman-tic concepts and low-level visual details. Notably, Chat-UniVi is trained on a mixed dataset containing both images and videos, allowing direct application to tasks involving both mediums without requiring any modifications. Exten-sive experimental results demonstrate that Chat- UniVi con-sistently outperforms even existing methods exclusively de-signed for either images or videos. Code is available at https://github.com/PKu-Yuan Group/Chat-UniVi.Keywords
Funding Information
- National Key R&D Program of China (2022ZD0l18101)
This publication has 18 references indexed in Scilit:
- DiffusionRet: Generative Text-Video Retrieval with Diffusion Model2023
- Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learning2023
- Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformer2022
- GroupViT: Semantic Segmentation Emerges from Text Supervision2022
- Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval2021
- GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering2019
- TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering2017
- Study on density peaks clustering based on k-nearest neighbors and principal component analysisKnowledge-Based Systems, 2016
- VQA: Visual Question Answering2015
- ActivityNet: A large-scale video benchmark for human activity understanding2015