PosterSum: A Multimodal Benchmark for Scientific Poster Summarization
Preprint
Abstract
Generating accurate and concise textual summaries from multimodal documents is challenging, especially when dealing with visually complex content like scientific posters. We introduce PosterSum, a novel benchmark to advance the development of vision-language models that can understand and summarize scientific posters into research paper abstracts. Our dataset contains 16,305 conference posters paired with their corresponding abstracts as summaries. Each poster is provided in image format and presents diverse visual understanding challenges, such as complex layouts, dense text regions, tables, and figures. We benchmark state-of-the-art Multimodal Large Language Models (MLLMs) on PosterSum and demonstrate that they struggle to accurately interpret and summarize scientific posters. We propose Segment & Summarize, a hierarchical method that outperforms current MLLMs on automated metrics, achieving a 3.14% gain in ROUGE-L. This will serve as a starting point for future research on poster summarization.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- MMBench: Is Your Multi-modal Model an All-Around Player?2024
- Improved Baselines with Visual Instruction Tuning2024
- LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding2024
- DocFormerv2: Local Features for Document UnderstandingProceedings of the AAAI Conference on Artificial Intelligence, 2024
- SlideVQA: A Dataset for Document Visual Question Answering on Multiple ImagesProceedings of the AAAI Conference on Artificial Intelligence, 2023
- Divide and Conquer: Answering Questions with Object Factorization and Compositional Reasoning2023
- SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in SummarizationTransactions of the Association for Computational Linguistics, 2022
- MMOCR2021
- TLDR: Extreme Summarization of Scientific Documents2020
- ScisummNet: A Large Annotated Corpus and Content-Impact Models for Scientific Paper Summarization with Citation NetworksProceedings of the AAAI Conference on Artificial Intelligence, 2019