GPT4Table: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study
Authors:
- Yuan SuiYSYuan Sui
- Mengyu ZhouMZMengyu Zhou
- Mingjie ZhouMZMingjie Zhou
- Shi HanSHShi Han
- Dongmei ZhangDZDongmei Zhang
Preprint
Abstract
Large language models (LLMs) are becoming attractive as few-shot reasoners to solve Natural Language (NL)-related tasks. However, there is still much to learn about how well LLMs understand structured data, such as tables. While it is true that tables can be used as inputs to LLMs with serialization, there lack of comprehensive studies examining whether LLMs can truly comprehend such data. In this paper, we try to understand this by designing a benchmark to evaluate the structural understanding capabilities (SUC) of LLMs. The benchmark we create includes seven tasks, each with its own unique challenges, \eg, cell lookup, row retrieval, and size detection. We run a series of evaluations on GPT-3.5 and GPT-4. We discover that the performance varied depending on a number of input choices, including table input format, content order, role prompting, and partition marks. Drawing from the insights gained through the benchmark evaluations, we then propose \textit{self-augmentation} for effective structural prompting, \eg, critical value / range identification using LLMs' internal knowledge. When combined with carefully chosen input choices, these structural prompting methods lead to promising improvements in LLM performance on a variety of tabular tasks, \eg, TabFact($\uparrow2.31\%$), HybridQA($\uparrow2.13\%$), SQA($\uparrow2.72\%$), Feverous($\uparrow0.84\%$), and ToTTo($\uparrow5.68\%$). We believe that our benchmark and proposed prompting methods can serve as a simple yet generic selection for future research. The code and data are released in \url{https://anonymous.4open.science/r/StructuredLLM-76F3}.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- UniSAr: a unified structure-aware autoregressive language model for text-to-SQL semantic parsingInternational Journal of Machine Learning and Cybernetics, 2023
- GitTables: A Large-Scale Corpus of Relational TablesProceedings of the ACM on Management of Data, 2023
- TableFormer: Table Structure Understanding with Transformers2022
- TUTA2021
- MATE: Multi-view Attention for Table Transformer Efficiency2021
- TableGPT: Few-shot Table-to-Text Generation with Table Structure Reconstruction and Content Matching2020
- HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data2020
- TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data2020
- TaPas: Weakly Supervised Table Parsing via Pre-training2020
- Web Table Extraction, Retrieval and Augmentation2019