V2C: Visual Voice Cloning
Authors:
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 21210-21219
Abstract
Existing Voice Cloning (VC) tasks aim to convert a para-graph text to a speech with desired voice specified by a ref-erence audio. This has significantly boosted the development of artificial speech applications. However, there also exist many scenarios that cannot be well reflected by these VC tasks, such as movie dubbing, which requires the speech to be with emotions consistent with the movie plots. To fill this gap, in this work we propose a new task named Vi-sual Voice Cloning (V2C), which seeks to convert a para-graph of text to a speech with both desired voice speci-fied by a reference audio and desired emotion specified by a reference video. To facilitate research in this field, we construct a dataset, V2C-Animation, and propose a strong baseline based on existing state-of-the-art (SoTA) VC techniques. Our dataset contains 10,217 animated movie clips covering a large variety of genres (e.g., Comedy, Fantasy) and emotions (e.g., happy, sad). We further design a set of evaluation metrics, named MCD-DTW-SL, which help eval-uate the similarity between ground-truth speeches and the synthesised ones. Extensive experimental results show that even SoTA VC methods cannot generate satisfying speeches for our V2C task. We hope the proposed new task together with the constructed dataset and evaluation metric will fa-cilitate the research in the field of voice cloning and broader vision-and-language community. Source code and dataset will be released in https://github.com/chenqi008/V2C.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Least Squares Generative Adversarial Networks2017
- Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi2017
- Lip Reading Sentences in the Wild2017
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Image-to-Image Translation with Conditional Adversarial Networks2017
- Deep Residual Learning for Image Recognition2016
- MovieQA: Understanding Stories in Movies through Question-Answering2016
- A dataset for Movie Description2015
- Speech Prosody in Speech Synthesis: Modeling and generation of prosody for high quality and flexible speech synthesis2015
- Dynamic Time Warping2007