Syntax Customized Video Captioning by Imitating Exemplar Sentences
- 30 November 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 44 (12) , 10209-10221
Abstract
Enhancing the diversity of sentences to describe video contents is an important problem arising in recent video captioning research. In this paper, we explore this problem from a novel perspective of customizing video captions by imitating exemplar sentence syntaxes. Specifically, given a video and any syntax-valid exemplar sentence, we introduce a new task of Syntax Customized Video Captioning (SCVC) aiming to generate one caption which not only semantically describes the video contents but also syntactically imitates the given exemplar sentence. To tackle the SCVC task, we propose a novel video captioning model, where a hierarchical sentence syntax encoder is first designed to extract the syntactic structure of the exemplar sentence, then a syntax conditioned caption decoder is devised to generate the syntactically structured caption expressing video semantics. As there is no available syntax customized groundtruth video captions, we tackle such a challenge by proposing a new training strategy, which leverages the traditional pairwise video captioning data and our collected exemplar sentences to accomplish the model learning. Extensive experiments, in terms of semantic, syntactic, fluency, and diversity evaluations, clearly demonstrate our model capability to generate syntax-varied and semantics-coherent video captions that well imitate different exemplar sentences with enriched diversities. Code is available at https://github.com/yytzsy/Syntax-Customized-Video-Captioning .Keywords
All Related Versions
Funding Information
- National Key Research and Development Program of China (2020AAA0106300, 2018AAA0102000)
- National Natural Science Foundation of China (62050110)
- Shenzhen Nanshan District Ling-Hang Team (LHTD20170005)
This publication has 33 references indexed in Scilit:
- Describing Videos by Exploiting Temporal Structure2015
- From captions to visual concepts and back2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Translating Videos to Natural Language Using Deep Recurrent Neural Networks2015
- Coherent Multi-sentence Video Description with Variable Level of Detail2014
- The Stanford CoreNLP Natural Language Processing Toolkit2014
- Less Grammar, More Features2014
- Glove: Global Vectors for Word Representation2014
- Translating Video Content to Natural Language Descriptions2013
- Simple Fast Algorithms for the Editing Distance between Trees and Related ProblemsSIAM Journal on Computing, 1989