Memory-QA: Answering Recall Questions Based on Multimodal Memories
Authors:
Preprint
Abstract
We introduce Memory-QA, a novel real-world task that involves answering recall questions about visual content from previously stored multimodal memories. This task poses unique challenges, including the creation of task-oriented memories, the effective utilization of temporal and location information within memories, and the ability to draw upon multiple memories to answer a recall question. To address these challenges, we propose a comprehensive pipeline, Pensieve, integrating memory-specific augmentation, time- and location-aware multi-signal retrieval, and multi-memory QA fine-tuning. We created a multimodal benchmark to illustrate various real challenges in this task, and show the superior performance of Pensieve over state-of-the-art solutions (up to 14% on QA accuracy).Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Retrieval-Augmented Multi-Modal Chain-of-Thoughts Reasoning for Large Language Models2025
- From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge2025
- Unifying Multimodal Retrieval via Document Screenshot Embedding2024
- SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLM2024
- MoqaGPT : Zero-Shot Multi-modal Open-domain Question Answering with Large Language Model2023
- How to Train Your Dragon: Diverse Augmentation Towards Generalizable Dense Retrieval2023
- Unified Language Representation for Question Answering over Text, Tables, and Images2023
- Retrieval Augmented Visual Question Answering with Outside Knowledge2022
- MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text2022
- Learning to Classify Text Using Support Vector Machines2002