DARTScore: DuAl-Reconstruction Transformer for Video Captioning Evaluation
- 31 July 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34 (4) , 2041-2055
Abstract
Video captioning evaluation aims at assessing the semantic consistency between video and candidate text, which should include measurement from two aspects: faithfulness (whether the information conveyed by candidate is correct w.r.t. video) and comprehensiveness (whether the main video content is covered by candidate). However, previous approaches have difficulty in evaluating faithfulness and comprehensiveness due to heavy reliance on references or heterogeneous of visual and textual data. In this paper, we propose a vision-involved evaluation metric based on a novel DuAl-Reconstruction Transformer, named DARTScore. DARTScore formulates the caption evaluation task as a dual-reconstruction problem to evaluate both faithfulness and comprehensiveness explicitly. Since the word in a candidate is usually related to several frames, DARTScore adaptively collects relevant frames to reconstruct the word and computes the reconstruction accuracy as faithfulness to inherently reflect whether the word information is contained in the video. In the inversive way, DARTScore reconstructs each frame with relevant words to evaluate comprehensiveness. By integrating fine-grained bidirectional reconstruction accuracies, DARTScore drills into each word in candidate and each frame in video to fully evaluate the semantic consistency. Furthermore, we collect and annotate two Chinese datasets with a large domain gap, named CRAETE-EVAL and VATEX-ZH-EVAL, to systematically evaluate existing metrics and fill the blank of Chinese video captioning evaluation. Experimental results show that DARTScore achieves higher correlation with human judgments, has lower reference reliance, and generalizes well to data from different domains.Keywords
Funding Information
- Beijing Natural Science Foundation (JQ21017)
- Natural Science Foundation of China (61972397, 62036011, 62192782, 61721004, U2033210)
- Guangdong Provincial University Innovation Team Project (2020KCXTD045)
This publication has 34 references indexed in Scilit:
- EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matching2022
- FAIEr: Fidelity and Adequacy Ensured Image Caption Evaluation2021
- Open-book Video Captioning with Retrieve-Copy-Generate Network2021
- VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research2019
- Reconstruction Network for Video Captioning2018
- Learning to Evaluate Image Captioning2018
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- SPICE: Semantic Propositional Image Caption Evaluation2016
- Deep Residual Learning for Image Recognition2016