TinyLVLM-eHub: Towards Comprehensive and Efficient Evaluation for Large Vision-Language Models
- 30 January 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Big Data
- Vol. 11 (3) , 933-947
Abstract
Large Vision-Language Models (LVLMs) have made significant strides in various multimodal tasks. Notably, GPT4V, Claude, Gemini, and others showcase exceptional multimodal capabilities, marked by profound comprehension and reasoning skills. This study introduces a comprehensive and efficient evaluation framework, TinyLVLM-eHub, to assess LVLMs’ performance, including proprietary models. TinyLVLM-eHub covers six key multimodal capabilities, such as visual perception, knowledge acquisition, reasoning, commonsense understanding, object hallucination, and embodied intelligence. The benchmark, utilizing 2.1K image-text pairs, provides a user-friendly and accessible platform for LVLM evaluation. The evaluation employs the ChatGPT Ensemble Evaluation (CEE) method, which improves alignment with human evaluation compared to word-matching approaches. Results reveal that closed-source API models like GPT4V and GeminiPro-V excel in most capabilities compared to previous open-source LVLMs, though they show some vulnerability in object hallucination. This evaluation underscores areas for LVLM improvement in real-world applications and serves as a foundational assessment for future multimodal advancements.Keywords
Funding Information
- National Key R & D Program of China (2022ZD0160101, 2022ZD0161000)
This publication has 32 references indexed in Scilit:
- OCRBench: on the hidden mystery of OCR in large multimodal modelsScience China Information Sciences, 2024
- MMBench: Is Your Multi-modal Model an All-Around Player?2024
- Toward Understanding WordArt: Corner-Guided Transformer for Scene Text Recognition2022
- Curved scene text detection via transverse and longitudinal sequence connectionPattern Recognition, 2019
- VirtualHome: Simulating Household Activities Via Programs2018
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- A robust arbitrary text detection system for natural scene imagesExpert Systems with Applications, 2014
- End-to-end scene text recognition using tree-structured modelsPattern Recognition, 2014
- Top-down and bottom-up cues for scene text recognition2012
- Cats and dogs2012