SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLM
Preprint
Abstract
Vision-extended LLMs have made significant strides in Visual Question Answering (VQA). Despite these advancements, VLLMs still encounter substantial difficulties in handling queries involving long-tail entities, with a tendency to produce erroneous or hallucinated responses. In this work, we introduce a novel evaluative benchmark named \textbf{SnapNTell}, specifically tailored for entity-centric VQA. This task aims to test the models' capabilities in identifying entities and providing detailed, entity-specific knowledge. We have developed the \textbf{SnapNTell Dataset}, distinct from traditional VQA datasets: (1) It encompasses a wide range of categorized entities, each represented by images and explicitly named in the answers; (2) It features QA pairs that require extensive knowledge for accurate responses. The dataset is organized into 22 major categories, containing 7,568 unique entities in total. For each entity, we curated 10 illustrative images and crafted 10 knowledge-intensive QA pairs. To address this novel task, we devised a scalable, efficient, and transparent retrieval-augmented multimodal LLM. Our approach markedly outperforms existing methods on the SnapNTell dataset, achieving a 66.5\% improvement in the BELURT score. We will soon make the dataset and the source code publicly accessible.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration2024
- Improved Baselines with Visual Instruction Tuning2024
- AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model2024
- Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?2024
- Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities2023
- Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?2023
- Billion-Scale Similarity Search with GPUsIEEE Transactions on Big Data, 2019
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question AnsweringInternational Journal of Computer Vision, 2018
- FVQA: Fact-Based Visual Question AnsweringIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Measuring nominal scale agreement among many raters.Psychological Bulletin, 1971