Maria: A Visual Experience Powered Conversational Agent
Preprint
Abstract
Arguably, the visual perception of conversational agents to the physical world is a key way for them to exhibit the human-like intelligence. Image-grounded conversation is thus proposed to address this challenge. Existing works focus on exploring the multimodal dialog models that ground the conversation on a given image. In this paper, we take a step further to study image-grounded conversation under a fully open-ended setting where no paired dialog and image are assumed available. Specifically, we present Maria, a neural conversation agent powered by the visual world experiences which are retrieved from a large-scale image index. Maria consists of three flexible components, i.e., text-to-image retriever, visual concept detector and visual-knowledge-grounded response generator. The retriever aims to retrieve a correlated image to the dialog from an image index, while the visual concept detector extracts rich visual knowledge from the image. Then, the response generator is grounded on the extracted visual knowledge and dialog context to generate the target response. Extensive experiments demonstrate Maria outperforms previous state-of-the-art methods on automatic metrics and human evaluation, and can generate informative responses that have some visual commonsense of the physical world.Keywords
All Related Versions
This publication has 30 references indexed in Scilit:
- MirrorGAN: Learning Text-To-Image Generation by Redescription2019
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks2018
- Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning2017
- Visual Dialog2017
- Aggregated Residual Transformations for Deep Neural Networks2017
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- End-to-End People Detection in Crowded Scenes2016
- Show and tell: A neural image caption generator2015
- The symbol grounding problemPhysica D: Nonlinear Phenomena, 1990