GPS–Relational: Generative Problem Solving with Relational Databases
- 21 July 2026
- journal article
- Published by World Scientific Pub Co Pte Ltd in International Journal of Semantic Computing
- Vol. 20 (03) , 395-419
Abstract
Large language models (LLMs) have improved natural language to SQL translation, but evaluating their outputs remains difficult when queries become complex, deeply nested, or produced at scale. Existing evaluations often rely on standard benchmarks with limited structural complexity or on model-based judgments that may be inconsistent. We present GPS-Relational, an evaluation framework that validates LLM-generated SQL by constructing an executable reference SQL query from the same natural-language intent and database schema. GPS-Relational builds this reference through grammar-guided decomposition, schema mapping, semantic validation, and SQL assembly. The reference SQL is executed to produce the expected result, which is then compared with the execution result of the LLM-generated SQL, allowing equivalent SQL forms to be accepted without requiring textual or structural matching. Additionally, GPS-Relational facilitates large-scale benchmark generation by systematically synthesizing extensive datasets of natural-language and SQL pairs with controlled nesting depth and relational complexity. These generated cases enable a systematic methodology to evaluate LLM performance across varied levels of structural complexity. Experimental results demonstrate that GPS-Relational constructs executable reference SQL across existing benchmark and complex generated queries, and that LLM execution accuracy degrades significantly as nesting depth increases up to 15 levels.Keywords
This publication has 16 references indexed in Scilit:
- Reliable Answers for Recurring Questions: Boosting Text-to-SQL Accuracy with Template Constrained DecodingProceedings of the ACM on Management of Data, 2025
- Bridging the Semantic Gap with SQL Query Logs in Natural Language Interfaces to Databases2019
- DBPal2018
- SQLizer: query synthesis from natural languageProceedings of the ACM on Programming Languages, 2017
- Synthesizing highly expressive SQL queries from input-output examples2017
- ATHENAProceedings of the VLDB Endowment, 2016
- NaLIR2014
- Automatically synthesizing SQL queries from input-output examples2013
- SODAProceedings of the VLDB Endowment, 2012
- Developing a natural language interface to complex dataACM Transactions on Database Systems, 1978