Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study
Top Cited Papers
Open Access
- 4 March 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 645-654
Abstract
Large language models (LLMs) are becoming attractive as few-shot reasoners to solve Natural Language (NL)-related tasks. However, there is still much to learn about how well LLMs understand structured data, such as tables. Although tables can be used as input to LLMs with serialization, there is a lack of comprehensive studies that examine whether LLMs can truly comprehend such data. In this paper, we try to understand this by designing a benchmark to evaluate the structural understanding capabilities (SUC) of LLMs. The benchmark we create includes seven tasks, each with its own unique challenges, \eg, cell lookup, row retrieval, and size detection. We perform a series of evaluations on GPT-3.5 and GPT-4. We find that performance varied depending on several input choices, including table input format, content order, role prompting, and partition marks. Drawing from the insights gained through the benchmark evaluations, we proposeself-augmentation for effective structural prompting, such as critical value / range identification using internal knowledge of LLMs. When combined with carefully chosen input choices, these structural prompting methods lead to promising improvements in LLM performance on a variety of tabular tasks, \eg, TabFact(\uparrow2.31%), HybridQA(\uparrow2.13%), SQA(\uparrow2.72%), Feverous(\uparrow0.84%), and ToTTo(\uparrow5.68%). We believe that our open-source (please find code and data at https://github.com/microsoft/TableProvider) benchmark and proposed prompting methods can serve as a simple yet generic selection for future research.Keywords
All Related Versions
This publication has 29 references indexed in Scilit:
- Simulating Users in Interactive Web Table Retrieval2023
- StruBERT: Structure-aware BERT for Table Search and Matching2022
- TUTA2021
- ToTTo: A Controlled Table-To-Text Generation DatasetarXiv, 2020
- TabFact: A Large-scale Dataset for Table-based Fact VerificationarXiv, 2020
- Web Table Extraction, Retrieval and Augmentation: A SurveyarXiv, 2020
- TableGPT: Few-shot Table-to-Text Generation with Table Structure Reconstruction and Content Matching2020
- HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data2020
- TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data2020
- Search-based Neural Structured Learning for Sequential Question Answering2017