Fast Capture of Cell-Level Provenance in Numpy
Preprint
Abstract
Effective provenance tracking enhances reproducibility, governance, and data quality in array workflows. However, significant challenges arise in capturing this provenance, including: (1) rapidly evolving APIs, (2) diverse operation types, and (3) large-scale datasets. To address these challenges, this paper presents a prototype annotation system designed for arrays, which captures cell-level provenance specifically within the numpy library. With this prototype, we explore straightforward memory optimizations that substantially reduce annotation latency. We envision this provenance capture approach for arrays as part of a broader governance system for tracking for structured data workflows and diverse data science applications.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Compression and In-Situ Query Processing for Fine-Grained Array Lineage2024
- Array programming with NumPyNature, 2020
- Towards scalable dataframe systemsProceedings of the VLDB Endowment, 2020
- Leakage in data miningACM Transactions on Knowledge Discovery From Data, 2012
- Data Structures for Statistical Computing in PythonProceedings of the Python in Science Conference, 2010
- The ORCHESTRA Collaborative Data Sharing SystemACM SIGMOD Record, 2008
- Provenance semirings2007