Tables as Images? Exploring the Strengths and Limitations of LLMs on Multimodal Representations of Tabular Data
Authors:
Preprint
Abstract
In this paper, we investigate the effectiveness of various LLMs in interpreting tabular data through different prompting strategies and data formats. Our analysis extends across six benchmarks for table-related tasks such as question-answering and fact-checking. We introduce for the first time the assessment of LLMs' performance on image-based table representations. Specifically, we compare five text-based and three image-based table representations, demonstrating the influence of representation and prompting on LLM performance. Our study provides insights into the effective use of LLMs on table-related tasks.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- What does CLIP know about a red circle? Visual prompt engineering for VLMs2023
- Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language ProcessingACM Computing Surveys, 2023
- TableFormer: Robust Transformer Modeling for Table-Text Encoding2022
- HittER: Hierarchical Transformers for Knowledge Graph Embeddings2021
- FinQA: A Dataset of Numerical Reasoning over Financial Data2021
- ToTTo: A Controlled Table-To-Text Generation Dataset2020
- Table Fact Verification with Structure-Aware Transformer2020
- TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data2020
- TaPas: Weakly Supervised Table Parsing via Pre-training2020
- Logical Natural Language Generation from Open-Domain Tables2020