Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities
Preprint
Abstract
Vision-language Models (VLMs) have emerged as general-purpose tools for addressing a variety of complex computer vision problems. Such models have been shown to be highly capable, but, at the same time, lacking some basic visual understanding skills. In this paper, we set out to understand the limitations of SoTA VLMs on fundamental visual tasks by constructing a series of tests that probe which components of design, specifically, may be lacking. Importantly, we go significantly beyond the current benchmarks, which simply measure the final performance of VLM response, by also comparing and contrasting it to the performance of probes trained directly on features obtained from the visual encoder, intermediate vision-language projection and LLM-decoder output. In doing so, we uncover shortcomings in VLMs and make a number of important observations about their capabilities, robustness and how they process visual information. We hope our insights will guide progress in further improving VLMs.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- Vision-Language Models Do Not Understand Negation2025
- VideoAgent: Long-Form Video Understanding with Large Language Model as Agent2024
- Probing the 3D Awareness of Visual Foundation Models2024
- Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding2024
- Talk2BEV: Language-enhanced Bird’s-eye View Maps for Autonomous Driving2024
- What does CLIP know about a red circle? Visual prompt engineering for VLMs2023
- What’s “up” with vision-language models? Investigating their struggle with spatial reasoning2023
- Richer Convolutional Features for Edge DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question AnsweringInternational Journal of Computer Vision, 2018
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009