Enhancing Multi-modal and Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generation
Preprint
Abstract
Multi-modal and multi-hop question answering aims to answer a question based on multiple input sources from different modalities. Previous methods retrieve the evidence separately and feed the retrieved evidence to a language model to generate the corresponding answer. However, these methods fail to build connections between candidates and thus cannot model the inter-dependent relation during retrieval. Moreover, the reasoning process over multi-modality candidates can be unbalanced without building alignments between different modalities. To address this limitation, we propose a Structured Knowledge and Unified Retrieval Generation based method (SKURG). We align the sources from different modalities via the shared entities and map them into a shared semantic space via structured knowledge. Then, we utilize a unified retrieval-generation decoder to integrate intermediate retrieval results for answer generation and adaptively determine the number of retrieval steps. We perform experiments on two multi-modal and multi-hop datasets: WebQA and MultimodalQA. The results demonstrate that SKURG achieves state-of-the-art performance on both retrieval and answer generation.Keywords
All Related Versions
- Version 1, 2022-12-16, ArXiv
- Version 2, 2023-08-07, ArXiv
- Published version: , 5223.
This publication has 6 references indexed in Scilit:
- ♫ MuSiQue: Multihop Questions via Single-hop Question CompositionTransactions of the Association for Computational Linguistics, 2022
- MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text2022
- UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answering2022
- MMCoQA: Conversational Question Answering over Text, Tables, and Images2022
- OpenNRE: An Open and Extensible Toolkit for Neural Relation Extraction2019
- Semi-supervised sequence tagging with bidirectional language models2017