Vision Generalist Model: A Survey
Authors:
- Ziyi WangZWZiyi Wang
- Yongming RaoYRYongming Rao
- Shuofeng SunSSShuofeng Sun
- Xinrun LiuXLXinrun Liu
- Yi WeiYWYi Wei
- Xumin YuXYXumin Yu
- Zuyan LiuZLZuyan Liu
- Yanbo WangYWYanbo Wang
- Hongmin LiuHLHongmin Liu
- Jie ZhouJZJie Zhou
- Jiwen LuJLJiwen Lu
Preprint
Abstract
Recently, we have witnessed the great success of the generalist model in natural language processing. The generalist model is a general framework trained with massive data and is able to process various downstream tasks simultaneously. Encouraged by their impressive performance, an increasing number of researchers are venturing into the realm of applying these models to computer vision tasks. However, the inputs and outputs of vision tasks are more diverse, and it is difficult to summarize them as a unified representation. In this paper, we provide a comprehensive overview of the vision generalist models, delving into their characteristics and capabilities within the field. First, we review the background, including the datasets, tasks, and benchmarks. Then, we dig into the design of frameworks that have been proposed in existing research, while also introducing the techniques employed to enhance their performance. To better help the researchers comprehend the area, we take a brief excursion into related domains, shedding light on their interconnections and potential synergies. To conclude, we provide some real-world application scenarios, undertake a thorough examination of the persistent challenges, and offer insights into possible directions for future research endeavors.Keywords
All Related Versions
This publication has 53 references indexed in Scilit:
- Graph Perceiver IO: A general architecture for graph-structured dataPattern Recognition, 2025
- GiT: Towards Generalist Vision Transformer Through Universal Language Interface2024
- PointLLM: Empowering Large Language Models to Understand Point Clouds2024
- MultiMAE: Multi-modal Multi-task Masked Autoencoders2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Taming Transformers for High-Resolution Image Synthesis2021
- Deepfake Video Detection Using Recurrent Neural Networks2018
- 3D Semantic Segmentation with Submanifold Sparse Convolutional Networks2018
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- Deep Residual Learning for Image Recognition2016