Revisiting Task-Oriented Dataset Search in the Era of Large Language Models: Challenges, Benchmark, and Solution
Authors:
Preprint
Abstract
The search for suitable datasets is the critical "first step" in data-driven research, but it remains a great challenge. Researchers often need to search for datasets based on high-level task descriptions. However, existing search systems struggle with this task due to ambiguous user intent, task-to-dataset mapping and benchmark gaps, and entity ambiguity. To address these challenges, we introduce KATS, a novel end-to-end system for task-oriented dataset search from unstructured scientific literature. KATS consists of two key components, i.e., offline knowledge base construction and online query processing. The sophisticated offline pipeline automatically constructs a high-quality, dynamically updatable task-dataset knowledge graph by employing a collaborative multi-agent framework for information extraction, thereby filling the task-to-dataset mapping gap. To further address the challenge of entity ambiguity, a unique semantic-based mechanism is used for task entity linking and dataset entity resolution. For online retrieval, KATS utilizes a specialized hybrid query engine that combines vector search with graph-based ranking to generate highly relevant results. Additionally, we introduce CS-TDS, a tailored benchmark suite for evaluating task-oriented dataset search systems, addressing the critical gap in standardized evaluation. Experiments on our benchmark suite show that KATS significantly outperforms state-of-the-art retrieval-augmented generation frameworks in both effectiveness and efficiency, providing a robust blueprint for the next generation of dataset discovery systems.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- The Faiss LibraryIEEE Transactions on Big Data, 2025
- SemExplorer: A User Interface for Semantic Approach to Customized Dataset Search2025
- Pneuma : Leveraging LLMs for Tabular Data Representation and Retrieval in an End-to-End SystemProceedings of the ACM on Management of Data, 2025
- It Took Longer than I was Expecting: Why is Dataset Search Still so Hard?2024
- Dataset Discovery and Exploration: A SurveyACM Computing Surveys, 2023
- Dataset Discovery in Data Lakes2020
- Dataset search: a surveyThe VLDB Journal, 2019
- Google Dataset Search: Building a search engine for datasets in an open Web ecosystem2019
- Dostoevsky2018
- Automated Quality Assessment of Metadata across Open Data PortalsJournal of Data and Information Quality, 2016