Show, Reward, and Tell
Open Access
- 30 April 2019
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 15 (2s) , 1-20
Abstract
Despite the promising progress made in visual captioning and paragraphing, visual storytelling is still largely unexplored. This task is more challenging due to the difficulty in modeling an ordered photo sequence and in generating a relevant paragraph with expressive language style for storytelling. To deal with these challenges, we propose an Attribute-based Hierarchical Generative model with Reinforcement Learning and adversarial training (AHGRL). First, to model the ordered photo sequence and the complex story structure, we propose an attribute-based hierarchical generator. The generator incorporates semantic attributes to create more accurate and relevant descriptions. The hierarchical framework enables the generator to learn from the complex paragraph structure. Second, to generate story-style paragraphs, we design a language-style discriminator, which provides word-level rewards to optimize the generator by policy gradient. Third, we further consider the story generator and the reward critic as adversaries. The generator aims to create indistinguishable paragraphs to human-level stories, whereas the critic aims at distinguishing them and further improving the generator. Extensive experiments on the widely used dataset well demonstrate the advantages of the proposed method over state-of-the-art methods.Keywords
This publication has 22 references indexed in Scilit:
- Generalized Deep Transfer Networks for Knowledge Propagation in Heterogeneous DomainsACM Transactions on Multimedia Computing, Communications, and Applications, 2016
- Tri-Clustered Tensor Completion for Social-Aware Image Tag RefinementIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Sequence to Sequence -- Video to Text2015
- Describing Videos by Exploiting Temporal Structure2015
- Long-term recurrent convolutional networks for visual recognition and description2015
- Show and tell: A neural image caption generator2015
- CIDEr: Consensus-based image description evaluation2015
- Baby talk: Understanding and generating simple image descriptions2011
- BLEU2001
- Simple statistical gradient-following algorithms for connectionist reinforcement learningMachine Learning, 1992