Distilling Answer-Set Programming Rules from LLMs for Neurosymbolic Visual Question Answering
Open Access
- 31 July 2026
- journal article
- research article
- Published by Cambridge University Press (CUP) in Theory and Practice of Logic Programming
- p. 1-35
Abstract
Visual Question Answering (VQA) is the task of answering questions about images, requiring the integration of multimodal input and reasoning. Modular approaches that incorporate logic-based representations into the reasoning component offer clear advantages over end-to-end trained systems, particularly in terms of interpretability. However, adapting or extending these representations when task requirements change can place a significant burden on developers. To address this challenge, we present an approach for distilling rules from Large Language Models (LLMs). Our method prompts an LLM to extend an initial VQA reasoning theory, expressed as an answer-set program, to meet new requirements of the task. Examples from VQA datasets guide the LLM, validate the results, and help correct erroneous rules by leveraging feedback from the ASP solver. We demonstrate that our approach is effective across diverse VQA datasets. Notably, only a few examples are needed to elicit correct rules from LLMs. Our experiments suggest that rule distillation from LLMs is a promising alternative to traditional data-driven rule learning approaches. Under consideration in Theory and Practice of Logic Programming. 1Keywords
All Related Versions
This publication has 45 references indexed in Scilit:
- Using Learning from Answer Sets for Robust Question Answering with LLM2024
- Simple Open-Vocabulary Object Detection2022
- Combining Deep Learning and ASP-Based Models for the Semantic Segmentation of Medical Images2021
- AQuA: ASP-Based Visual Question Answering2020
- Answer Set Programming2019
- Statistical Relational Learning2017
- Optical Graph RecognitionLecture Notes in Computer Science, 2013
- Logic Programming and Nonmonotonic Reasoning2013
- TildeCRF: Conditional Random Fields for Logical SequencesLecture Notes in Computer Science, 2006
- Recursive Aggregates in Disjunctive Logic Programs: Semantics and Complexity2004