SPARQL-LLM: Real-Time SPARQL Query Generation from Natural Language Questions
Preprint
Abstract
The advent of large language models is contributing to the emergence of novel approaches that promise to better tackle the challenge of generating structured queries, such as SPARQL queries, from natural language. However, these new approaches mostly focus on response accuracy over a single source while ignoring other evaluation criteria, such as federated query capability over distributed data stores, as well as runtime and cost to generate SPARQL queries. Consequently, they are often not production-ready or easy to deploy over (potentially federated) knowledge graphs with good accuracy. To mitigate these issues, in this paper, we extend our previous work and describe and systematically evaluate SPARQL-LLM, an open-source and triplestore-agnostic approach, powered by lightweight metadata, that generates SPARQL queries from natural language text. First, we describe its architecture, which consists of dedicated components for metadata indexing, prompt building, and query generation and execution. Then, we evaluate it based on a state-of-the-art challenge with multilingual questions, and a collection of questions from three of the most prevalent knowledge graphs within the field of bioinformatics. Our results demonstrate a substantial increase of 24% in the F1 Score on the state-of-the-art challenge, adaptability to high-resource languages such as English and Spanish, as well as ability to form complex and federated bioinformatics queries. Furthermore, we show that SPARQL-LLM is up to 36x faster than other systems participating in the challenge, while costing a maximum of $0.01 per question, making it suitable for real-time, low-cost text-to-SPARQL applications. One such application deployed over real-world decentralized knowledge graphs can be found at https://www.expasy.org/chat.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- GRASP: Generic Reasoning And SPARQL Generation Across Knowledge Graphs2025
- SPARQL Query Generation with LLMs: Measuring the Impact of Training Data Memorization and Knowledge Injection2025
- A large collection of bioinformatics question–query pairs over federated knowledge graphs: methodology and applicationsGigaScience, 2025
- UniProt: the Universal Protein Knowledgebase in 2025Nucleic Acids Research, 2024
- The SIB Swiss Institute of Bioinformatics Semantic Web of dataNucleic Acids Research, 2023
- QALD-9-plus: A Multilingual Dataset for Question Answering over DBpedia and Wikidata Translated by Native Speakers2022
- The Bgee suite: integrated curated expression atlas and comparative transcriptomics in animalsNucleic Acids Research, 2020
- LC-QuAD: A Corpus for Complex Question Answering over Knowledge Graphs2017
- Core techniques of question answering systems over knowledge bases: a surveyKnowledge and Information Systems, 2017
- SPARQL Assist language-neutral query composerBMC Bioinformatics, 2012