Open Access
- 1 December 2018
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in Proceedings of the VLDB Endowment
- Vol. 12 (4) , 419-432
Abstract
A range of explanation engines assist data analysts by performing feature selection over increasingly high-volume and high-dimensional data, grouping and highlighting commonalities among data points. While useful in diverse tasks such as user behavior analytics, operational event processing, and root cause analysis, today's explanation engines are designed as standalone data processing tools that do not interoperate with traditional, SQL-based analytics workflows; this limits the applicability and extensibility of these engines. In response, we propose the DIFF operator, a relational aggregation operator that unifies the core functionality of these engines with declarative relational query processing. We implement both single-node and distributed versions of the DIFF operator in MB SQL, an extension of MacroBase, and demonstrate how DIFF can provide the same semantics as existing explanation engines while capturing a broad set of production use cases in industry, including at Microsoft and Facebook. Additionally, we illustrate how this declarative approach to data explanation enables new logical and physical query optimizations. We evaluate these optimizations on several real-world production applications, and find that DIFF in MB SQL can outperform state-of-the-art engines by up to an order of magnitude.Keywords
This publication has 49 references indexed in Scilit:
- The tail at scaleCommunications of the ACM, 2013
- DremelProceedings of the VLDB Endowment, 2010
- The cost of a cloudACM SIGCOMM Computer Communication Review, 2008
- Architecture of a Database SystemFoundations and Trends® in Databases, 2007
- A review of feature selection techniques in bioinformaticsBioinformatics, 2007
- SQL Based Frequent Pattern Mining with FP-GrowthLecture Notes in Computer Science, 2005
- EddiesACM SIGMOD Record, 2000
- Improved query performance with variant indexesACM SIGMOD Record, 1997
- Applications of Range Query Theory to Relational Data Base Join and Selection OperationsJournal of Computer and System Sciences, 1996
- On the propagation of errors in the size of join resultsACM SIGMOD Record, 1991