Audiopedia: Audio QA with Knowledge
Authors:
- 6 April 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-5
Abstract
In this paper, we introduce Audiopedia, a novel task called Audio Question Answering with Knowledge, which requires both audio comprehension and external knowledge reasoning. Unlike traditional Audio Question Answering (AQA) benchmarks that focus on simple queries answerable from audio alone, Audiopedia targets knowledge-intensive questions. We define three sub-tasks: (i) Single Audio Question Answering (s-AQA), where questions are answered based on a single audio sample, (ii) Multi-Audio Question Answering (m-AQA), which requires reasoning over multiple audio samples, and (iii) Retrieval-Augmented Audio Question Answering (r-AQA), which involves retrieving relevant audio to answer the question. We benchmark large audio language models (LALMs) on these sub-tasks and observe suboptimal performance. To address this, we propose a generic framework that can be adapted to any LALM, equipping them with knowledge reasoning capabilities. Our framework has two components: (i) Audio Entity Linking (AEL) and (ii) Knowledge-Augmented Audio Large Multimodal Model (KA2LM), which together improve performance on knowledge-intensive AQA tasks. To our knowledge, this is the first work to address advanced speech audio understanding through knowledge-intensive tasks like Audiopedia. We make our data publicly available: https://abhiram4572.github.io/projects/audiopedia/.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- Joint Audio and Speech Understanding2023
- A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge2022
- Recent Advances in Video Question Answering: A Review of Datasets and Methods2021
- Visual Question Answering on Image Sets2020
- From Strings to Things: Knowledge-Enabled VQA Model That Can Read and Reason2019
- Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions2018
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning2017
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering2017
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- VQA: Visual Question Answering2015