MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation
Preprint
Abstract
Responding with multi-modal content has been recognized as an essential capability for an intelligent conversational agent. In this paper, we introduce the MMDialog dataset to better facilitate multi-modal conversation. MMDialog is composed of a curated set of 1.08 million real-world dialogues with 1.53 million unique images across 4,184 topics. MMDialog has two main and unique advantages. First, it is the largest multi-modal conversation dataset by the number of dialogues by 8x. Second, it contains massive topics to generalize the open-domain. To build engaging dialogue system with this dataset, we propose and normalize two response producing tasks based on retrieval and generative scenarios. In addition, we build two baselines for above tasks with state-of-the-art techniques and report their experimental performance. We also propose a novel evaluation metric MM-Relevance to measure the multi-modal responses. Our dataset and scripts are available in https://github.com/victorsungo/MMDialog.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Small Changes Make Big Differences: Improving Multi-turn Response Selection in Dialogue Systems via Fine-Grained Contrastive Learning2022
- Multimodal Dialogue Response Generation2022
- Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO2021
- Constructing Multi-Modal Dialogue Dataset by Replacing Text with Semantically Relevant Images2021
- PhotoChat: A Human-Human Dialogue Dataset With Photo Sharing Behavior For Joint Image-Text Modeling2021
- Maria: A Visual Experience Powered Conversational Agent2021
- Image-Chat: Engaging Grounded Conversations2020
- Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog2019
- Adversarial Learning for Neural Dialogue Generation2017
- Modeling Context in Referring Expressions2016