Towards a Robust Retrieval-Based Summarization System
Preprint
Abstract
This paper describes an investigation of the robustness of large language models (LLMs) for retrieval augmented generation (RAG)-based summarization tasks. While LLMs provide summarization capabilities, their performance in complex, real-world scenarios remains under-explored. Our first contribution is LogicSumm, an innovative evaluation framework incorporating realistic scenarios to assess LLM robustness during RAG-based summarization. Based on limitations identified by LogiSumm, we then developed SummRAG, a comprehensive system to create training dialogues and fine-tune a model to enhance robustness within LogicSumm's scenarios. SummRAG is an example of our goal of defining structured methods to test the capabilities of an LLM, rather than addressing issues in a one-off fashion. Experimental results confirm the power of SummRAG, showcasing improved logical coherence and summarization quality. Data, corresponding model weights, and Python code are available online.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- List-aware Reranking-Truncation Joint Model for Search and Retrieval-augmented Generation2024
- Benchmarking Large Language Models in Retrieval-Augmented GenerationProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Benchmarking Large Language Models for News SummarizationTransactions of the Association for Computational Linguistics, 2024
- Evaluating large language models on medical evidence summarizationnpj Digital Medicine, 2023
- BRIO: Bringing Order to Abstractive Summarization2022
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering2021
- Dense Passage Retrieval for Open-Domain Question Answering2020
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- Text Summarization with Pretrained Encoders2019
- Two uses of anaphora resolution in summarizationInformation Processing & Management, 2007