Language-Based Image Editing with Recurrent Attentive Models
Authors:
- Jianbo ChenJCJianbo Chen
- Yelong ShenYSYelong Shen
- Jianfeng GaoJGJianfeng Gao
- Jingjing LiuJLJingjing Liu
- Xiaodong LiuXLXiaodong Liu
- 1 June 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8721-8729
Abstract
We investigate the problem of Language-Based Image Editing (LBIE). Given a source image and a natural language description, we want to generate a target image by editing the source image based on the description. We propose a generic modeling framework for two subtasks of LBIE: language-based image segmentation and image colorization. The framework uses recurrent attentive models to fuse image and language features. Instead of using a fixed step size, we introduce for each region of the image a termination gate to dynamically determine after each inference step whether to continue extrapolating additional information from the textual description. The effectiveness of the framework is validated on three datasets. First, we introduce a synthetic dataset, called CoSaL, to evaluate the end-to-end performance of our LBIE system. Second, we show that the framework leads to state-of-the-art performance on image segmentation on the ReferIt dataset. Third, we present the first language-based colorization result on the Oxford-102 Flowers dataset.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Colorful Image Colorization2016
- Natural Language Object Retrieval2016
- Stacked Attention Networks for Image Question Answering2016
- Deep Colorization2015
- VQA: Visual Question Answering2015
- Long-term recurrent convolutional networks for visual recognition and description2015
- Evaluation of output embeddings for fine-grained image classification2015
- Deep visual-semantic alignments for generating image descriptions2015
- Show and tell: A neural image caption generator2015
- Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning1992