Can Audio Captions Be Evaluated With Image Caption Metrics?
Authors:
- 23 May 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 981-985
Abstract
Automated audio captioning aims at generating textual descriptions for an audio clip. To evaluate the quality of generated audio captions, previous works directly adopt image captioning metrics like SPICE and CIDEr, without justifying their suitability in this new domain, which may mislead the development of advanced models. This problem is still unstudied due to the lack of human judgment datasets on caption quality. Therefore, we first construct two evaluation benchmarks, AudioCaps-Eval and Clotho-Eval. They are established with pairwise comparison instead of absolute rating to achieve better inter-annotator agreement. Current metrics are found in poor correlation with human annotations on these datasets. To overcome their limitations, we propose a metric named FENSE, where we combine the strength of Sentence-BERT in capturing similarity, and a novel Error Detector to penalize erroneous sentences for robustness. On the newly established benchmarks, FENSE outperforms current metrics by 14-25% accuracy. 1Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Clotho: an Audio Captioning Dataset2020
- BLEURT: Learning Robust Metrics for Text Generation2020
- Audio Caption: Listen and Tell2019
- TIGEr: Text-to-Image Grounding for Image Caption Evaluation2019
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks2019
- The price of debiasing automatic metrics in natural language evalaution2018
- Automated audio captioning with recurrent neural networks2017
- Why We Need New Evaluation Metrics for NLG2017
- CIDEr: Consensus-based image description evaluation2015
- Measuring nominal scale agreement among many raters.Psychological Bulletin, 1971