Tables as Images? Exploring the Strengths and Limitations of LLMs on Multimodal Representations of Tabular Data
Authors:
Preprint
Abstract
In this paper, we investigate the effectiveness of various LLMs in interpreting tabular data through different prompting strategies and data formats. Our analysis extends across six benchmarks for table-related tasks such as question-answering and fact-checking. We introduce for the first time the assessment of LLMs' performance on image-based table representations. Specifically, we compare five text-based and three image-based table representations, demonstrating the influence of representation and prompting on LLM performance. Our study provides insights into the effective use of LLMs on table-related tasks.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- What does CLIP know about a red circle? Visual prompt engineering for VLMs2023
- Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language ProcessingACM Computing Surveys, 2023
- A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity2023
- Hi-ToM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models2023
- Reasoning with Language Model Prompting: A Survey2023
- HittER: Hierarchical Transformers for Knowledge Graph Embeddings2021
- FinQA: A Dataset of Numerical Reasoning over Financial Data2021
- ToTTo: A Controlled Table-To-Text Generation Dataset2020
- Logical Natural Language Generation from Open-Domain Tables2020
- Compositional Semantic Parsing on Semi-Structured Tables2015