Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization
Authors:
- Yan LiYLYan Li
Preprint
Abstract
Text summarization has been a crucial problem in natural language processing (NLP) for several decades. It aims to condense lengthy documents into shorter versions while retaining the most critical information. Various methods have been proposed for text summarization, including extractive and abstractive summarization. The emergence of large language models (LLMs) like GPT3 and ChatGPT has recently created significant interest in using these models for text summarization tasks. Recent studies \cite{goyal2022news, zhang2023benchmarking} have shown that LLMs-generated news summaries are already on par with humans. However, the performance of LLMs for more practical applications like aspect or query-based summaries is underexplored. To fill this gap, we conducted an evaluation of ChatGPT's performance on four widely used benchmark datasets, encompassing diverse summaries from Reddit posts, news articles, dialogue meetings, and stories. Our experiments reveal that ChatGPT's performance is comparable to traditional fine-tuning methods in terms of Rouge scores. Moreover, we highlight some unique differences between ChatGPT-generated summaries and human references, providing valuable insights into the superpower of ChatGPT for diverse text summarization tasks. Our findings call for new directions in this area, and we plan to conduct further research to systematically examine the characteristics of ChatGPT-generated summaries through extensive human evaluation.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Benchmarking Large Language Models for News SummarizationTransactions of the Association for Computational Linguistics, 2024
- A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity2023
- OASum: Large-Scale Open Domain Aspect-based Summarization2023
- NEWTS: A Corpus for News Topic-Focused Summarization2022
- QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization2021
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization2019
- Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization2018
- Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies2018