Benchmark Evaluations, Applications, and Challenges of Large Vision Language Models: A Survey
Preprint
Abstract
Multimodal Vision Language Models (VLMs) have emerged as a transformative technology at the intersection of computer vision and natural language processing, enabling machines to perceive and reason about the world through both visual and textual modalities. For example, models such as CLIP, Claude, and GPT-4V demonstrate strong reasoning and understanding abilities on visual and textual data and beat classical single modality vision models on zero-shot classification. Despite their rapid advancements in research and growing popularity in applications, a comprehensive survey of existing studies on VLMs is notably lacking, particularly for researchers aiming to leverage VLMs in their specific domains. To this end, we provide a systematic overview of VLMs in the following aspects: model information of the major VLMs developed over the past five years (2019-2024); the main architectures and training methods of these VLMs; summary and categorization of the popular benchmarks and evaluation metrics of VLMs; the applications of VLMs including embodied agents, robotics, and video generation; the challenges and issues faced by current VLMs such as hallucination, fairness, and safety. Detailed collections including papers and model repository links are listed in https://github.com/zli12321/Awesome-VLM-Papers-And-Models.git.Keywords
All Related Versions
This publication has 84 references indexed in Scilit:
- Towards Multimodal In-context Learning for Vision and Language Models2025
- Images are Achilles’ Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models2024
- MMBench: Is Your Multi-modal Model an All-Around Player?2024
- Octopus: Embodied Vision-Language Programmer from Environmental Feedback2024
- A survey on knowledge-enhanced multimodal learningArtificial Intelligence Review, 2024
- M-FLAG: Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimization2023
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question AnsweringInternational Journal of Computer Vision, 2018
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- VQA: Visual Question AnsweringInternational Journal of Computer Vision, 2016
- A Diagram is Worth a Dozen Images2016