Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs
Authors:
- 17 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1818-1826
Abstract
Multimodal LLMs are the natural evolution of LLMs, and enlarge their capabilities so as to work beyond the pure textual modality. As research is being carried out to design novel architectures and vision-and-language adapters, in this paper we concentrate on endowing such models with the capability of answering questions that require external knowledge. Our approach, termed Wiki-LLaVA, aims at integrating an external knowledge source of multimodal documents, which is accessed through a hierarchical retrieval pipeline. Relevant passages, using this approach, are retrieved from the external knowledge source and employed as additional context for the LLM, augmenting the effectiveness and precision of generated dialogues. We conduct extensive experiments on datasets tailored for visual question answering with external data and demonstrate the appropriateness of our approach.Keywords
All Related Versions
Funding Information
- European Commission
This publication has 20 references indexed in Scilit:
- Cross-Modal Retrieval for Knowledge-Based Visual Question Answering2024
- Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation2023
- Reveal: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory2023
- Smallcap: Lightweight Image Captioning Prompted with Retrieval Augmentation2023
- A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge2022
- VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning2022
- Benchmarking Representation Learning for Natural World Image Collections2021
- Google Landmarks Dataset v2 – A Large-Scale Benchmark for Instance-Level Recognition and Retrieval2020
- OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge2019
- blip2011