Beyond Lexical Overlap: Validating Semantic Similarity Metrics for LLM-Based Video Caption Augmentation
Authors:
- Juvenal Francisco BarajasJBJuvenal Francisco Barajas
- Gio JungGJGio Jung
- Manali SethMSManali Seth
- Lana DoLDLana Do
- Malleeswari JagabattuniMJMalleeswari Jagabattuni
- Andrew Taylor ScottASAndrew Taylor Scott
- Shasta IhornSIShasta Ihorn
- Alexander Mario BlumABAlexander Mario Blum
- Vassilis AthitsosVAVassilis Athitsos
- Ilmi YoonIYIlmi Yoon
Open Access
- 5 March 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 166-172
Abstract
Large language models (LLMs) can generate synthetic video captions at scale, offering a practical way to enrich datasets that lack multiple human references. However, evaluating the semantic reliability of such captions is challenging, particularly when traditional lexical-overlap metrics (e.g., BLEU, ROUGE, METEOR, CIDEr, SPICE) are sensitive to surface-level wording and often misalign with human judgments. This paper investigates whether sentence embedding models provide more reliable evaluation metrics for LLM-generated video captions. We first compare five embedding models—Sentence-BERT, SimCSE, MPNet, and two GPT-based embeddings—against traditional lexical metrics using 450 caption pairs from VATEX, a benchmark video captioning dataset. Through validation with a linguistic specialist, we find that sentence embedding-based approaches achieve substantially higher agreement with expert judgments using Kendall’s τ correlation. We then used GPT-4o to generate ten synthetic captions per video from VATEX and apply the validated embedding metrics to compare semantic similarities with the human-human caption pairs. We applied the same pipeline to YuWa, a single-reference accessibility video dataset, which demonstrates a real-world use case. Our findings suggest that synthetic captions achieve more consistency in semantic similarity levels compared to independently written human captions. Notably, augmenting YuWa’s single human caption per video with ten LLM-generated captions produces semantic similarity scores matching or surpassing those of VATEX’s ten independently written human captions. This work provides the first expert-validated framework for evaluating LLM-generated video captions and demonstrates a practical pipeline for quality-controlled dataset augmentation in accessibility research.Keywords
This publication has 12 references indexed in Scilit:
- Enhancing video temporal grounding with large language model-based data augmentationThe Journal of Supercomputing, 2025
- Data Augmentation using LLMs: Data Perspectives, Learning Paradigms and Challenges2024
- VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research2019
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks2019
- Dense-Captioning Events in Videos2017
- Re-evaluating Automatic Metrics for Image Captioning2017
- SPICE: Semantic Propositional Image Caption Evaluation2016
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language2016
- CIDEr: Consensus-based image description evaluation2015
- A NEW MEASURE OF RANK CORRELATIONBiometrika, 1938