Benchmarking Large Language Models for Conversational Question Answering in Multi-instructional Documents
Preprint
Abstract
Instructional documents are rich sources of knowledge for completing various tasks, yet their unique challenges in conversational question answering (CQA) have not been thoroughly explored. Existing benchmarks have primarily focused on basic factual question-answering from single narrative documents, making them inadequate for assessing a model`s ability to comprehend complex real-world instructional documents and provide accurate step-by-step guidance in daily life. To bridge this gap, we present InsCoQA, a novel benchmark tailored for evaluating large language models (LLMs) in the context of CQA with instructional documents. Sourced from extensive, encyclopedia-style instructional content, InsCoQA assesses models on their ability to retrieve, interpret, and accurately summarize procedural guidance from multiple documents, reflecting the intricate and multi-faceted nature of real-world instructional tasks. Additionally, to comprehensively assess state-of-the-art LLMs on the InsCoQA benchmark, we propose InsEval, an LLM-assisted evaluator that measures the integrity and accuracy of generated responses and procedural instructions.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Improved Baselines with Visual Instruction Tuning2024
- MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning2024
- Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactions2024
- AssistQ: Affordance-Centric Question-Driven Task Completion for Egocentric Assistant2022
- MMCoQA: Conversational Question Answering over Text, Tables, and Images2022
- Question Rewriting for Conversational Question Answering2021
- Open-Domain Question Answering Goes Conversational via Question Rewriting2021
- Look before you Hop: Conversational Question Answering over Knowledge Graphs Using Judicious Context Expansion2019
- CoQA: A Conversational Question Answering ChallengeTransactions of the Association for Computational Linguistics, 2019
- COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis2019