TENSAI - Practical and Responsible Observability for Data Quality-aware Large-scale Analytics
Open Access
- 19 December 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in Journal of Data and Information Quality
- Vol. 16 (4) , 1-43
Abstract
Given a large-scale mobile network with a variety of equipment and radio access network technologies for an approximate 20 million subscribers, there are many types of data that can be used for big data analytics and machine learning (ML) tasks for network operations, monitoring, and optimization. However, a variety of data is measured, collected, and propagated through numerous complex data and software systems. Thus, people, software components, and data-driven operations for big data and ML pipelines face great challenges in dealing with data quality impacts. Data quality related problems occur and are propagated through complex operations involving different types of data, people, software components, and analytics that cannot be solved purely through data quality engineering. This article discusses our TENSAI framework, as a practical and responsible observability for ensuring data quality in such a mobile network. TENSAI focuses on methods of communication, strategy specifications, and data quality engineering for diverse types of data and analytics among different types of operations. TENSAI presents techniques for capturing and communicating causes/effects of data quality problems clearly to all relevant stakeholders, developing data quality-aware adaptation strategies for actions on data that can be integrated into analytics processes, and engineering data quality awareness in software and data pipelines. Thus, TENSAI supports full visibility of data quality problems and impacts among related systems to empower the utilization and adaptation of data analytics for different types of operations. We will illustrate our TENSAI with several real-world data types, pipelines, and cases based on a real-world mobile network.Keywords
This publication has 44 references indexed in Scilit:
- Enjoy your observability: an industrial survey of microservice tracing and analysisEmpirical Software Engineering, 2021
- The
t
-digest: Efficient estimates of distributionsSoftware Impacts, 2021
- A hybrid method of exponential smoothing and recurrent neural networks for time series forecastingInternational Journal of Forecasting, 2020
- Machine learning based concept drift detection for predictive maintenanceComputers & Industrial Engineering, 2019
- Deep learning for time series classification: a reviewData Mining and Knowledge Discovery, 2019
- A recent overview of the state-of-the-art elements of text classificationExpert Systems with Applications, 2018
- Survey on time series motif discoveryWIREs Data Mining and Knowledge Discovery, 2017
- A Data Quality in Use model for Big DataFuture Generation Computer Systems, 2016
- A survey on data stream clustering and classificationKnowledge and Information Systems, 2014
- Time series motif discovery: dimensions and applicationsWIREs Data Mining and Knowledge Discovery, 2014