CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
Authors:
- Kanzhi ChengKCKanzhi Cheng
- Wenpo SongWSWenpo Song
- Jiaxin FanJFJiaxin Fan
- Zheng MaZMZheng Ma
- Qiushi SunQSQiushi Sun
- Fangzhi XuFXFangzhi Xu
- Chenyang YanCYChenyang Yan
- Nuo ChenNCNuo Chen
- Jianbing ZhangJZJianbing Zhang
- Jiajun ChenJCJiajun Chen
Preprint
Abstract
Image captioning has been a longstanding challenge in vision-language research. With the rise of LLMs, modern Vision-Language Models (VLMs) generate detailed and comprehensive image descriptions. However, benchmarking the quality of such captions remains unresolved. This paper addresses two key questions: (1) How well do current VLMs actually perform on image captioning, particularly compared to humans? We built CapArena, a platform with over 6000 pairwise caption battles and high-quality human preference votes. Our arena-style evaluation marks a milestone, showing that leading models like GPT-4o achieve or even surpass human performance, while most open-source models lag behind. (2) Can automated metrics reliably assess detailed caption quality? Using human annotations from CapArena, we evaluate traditional and recent captioning metrics, as well as VLM-as-a-Judge. Our analysis reveals that while some metrics (e.g., METEOR) show decent caption-level agreement with humans, their systematic biases lead to inconsistencies in model ranking. In contrast, VLM-as-a-Judge demonstrates robust discernment at both the caption and model levels. Building on these insights, we release CapArena-Auto, an accurate and efficient automated benchmark for detailed captioning, achieving 94.3% correlation with human rankings at just $4 per test. Data and resources will be open-sourced at https://caparena.github.io.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- LLLaVA-Critic: Learning to Evaluate Multimodal Models2025
- VisionArena: 230K Real World User-VLM Conversations with Preference Labels2025
- Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning2025
- OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis2025
- FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model2024
- SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents2024
- Evaluating Object Hallucination in Large Vision-Language Models2023
- Transparent Human Evaluation for Image Captioning2022
- CLIPScore: A Reference-free Evaluation Metric for Image Captioning2021
- Framing Image Description as a Ranking Task: Data, Models and Evaluation MetricsJournal of Artificial Intelligence Research, 2013