RELATE: Subjective evaluation dataset for automatic evaluation of relevance between text and audio
Preprint
Abstract
In text-to-audio (TTA) research, the relevance between input text and output audio is an important evaluation aspect. Traditionally, it has been evaluated from both subjective and objective perspectives. However, subjective evaluation is costly in terms of money and time, and objective evaluation is unclear regarding the correlation to subjective evaluation scores. In this study, we construct RELATE, an open-sourced dataset that subjectively evaluates the relevance. Also, we benchmark a model for automatically predicting the subjective evaluation score from synthesized audio. Our model outperforms a conventional CLAPScore model, and that trend extends to many sound categories.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- PAM: Prompting Audio-Language Models for Audio Quality Assessment2024
- AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised PretrainingIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024
- Diffsound: Discrete Diffusion Model for Text-to-Sound GenerationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- BYOL for Audio: Exploring Pre-Trained General-Purpose Audio RepresentationsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022
- UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 20222022
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech ProcessingIEEE Journal of Selected Topics in Signal Processing, 2022
- A Multiple Comparison Rank Sum Test: Treatments versus ControlBiometrics, 1959
- A new readability yardstick.Journal of Applied Psychology, 1948