Visually Interpretable Subtask Reasoning for Visual Question Answering
Preprint
Abstract
Answering complex visual questions like `Which red furniture can be used for sitting?' requires multi-step reasoning, including object recognition, attribute filtering, and relational understanding. Recent work improves interpretability in multimodal large language models (MLLMs) by decomposing tasks into sub-task programs, but these methods are computationally expensive and less accurate due to poor adaptation to target data. To address this, we introduce VISTAR (Visually Interpretable Subtask-Aware Reasoning Model), a subtask-driven training framework that enhances both interpretability and reasoning by generating textual and visual explanations within MLLMs. Instead of relying on external models, VISTAR fine-tunes MLLMs to produce structured Subtask-of-Thought rationales (step-by-step reasoning sequences). Experiments on two benchmarks show that VISTAR consistently improves reasoning accuracy while maintaining interpretability. Our code and dataset will be available at https://github.com/ChengJade/VISTAR.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- HYDRA: A Hyper Agent for Dynamic Compositional Visual Reasoning2024
- Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models2024
- VTimeLLM: Empower LLM to Grasp Video Moments2024
- Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models2024
- Visual Programming: Compositional visual reasoning without training2023
- Grounded Language-Image Pre-training2022
- Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression2019
- GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering2019
- Towards VQA Models That Can Read2019
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question AnsweringInternational Journal of Computer Vision, 2018