Metadata Management for AI-Augmented Data Workflows
Preprint
Abstract
AI-augmented data workflows introduce complex governance challenges, as both human and model-driven processes generate, transform, and consume data artifacts. These workflows blend heterogeneous tools, dynamic execution patterns, and opaque model decisions, making comprehensive metadata capture difficult. In this work, we present TableVault, a metadata governance framework designed for human-AI collaborative data creation. TableVault records ingestion events, traces operation status, links execution parameters to their data origins, and exposes a standardized metadata layer. By combining database-inspired guarantees with AI-oriented design, such as declarative operation builders and lineage-aware references, TableVault supports transparency and reproducibility across mixed human-model pipelines. Through a document classification case study, we demonstrate how TableVault preserves detailed lineage and operational context, enabling robust metadata management, even in partially observable execution environments.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- DocETL: Agentic Query Rewriting and Evaluation for Complex Document ProcessingProceedings of the VLDB Endowment, 2025
- Compression and In-Situ Query Processing for Fine-Grained Array Lineage2024
- Large Language Models Enable Few-Shot ClusteringTransactions of the Association for Computational Linguistics, 2024
- Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer reviewResearch Integrity and Peer Review, 2023
- ClusterLLM: Large Language Models as a Guide for Text Clustering2023
- Bao: Making Learned Query Optimization PracticalACM SIGMOD Record, 2021
- Array programming with NumPyNature, 2020
- Towards scalable dataframe systemsProceedings of the VLDB Endowment, 2020
- The FAIR Guiding Principles for scientific data management and stewardshipScientific Data, 2016
- Jupyter Notebooks – a publishing format for reproducible computational workflows2016