Intelligent Identification and Order-Sensitive Correction Method of Outliers from Multi-Data Source Based on Historical Data Mining
Open Access
- 7 September 2022
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 11 (18) , 2819
Abstract
In recent years, outliers caused by manual operation errors and equipment acquisition failures often occur, bringing challenges to big data analysis. In view of the difficulties in identifying and correcting outliers of multi-source data, an intelligent identification and order-sensitive correction method of outliers from multi-data sources based on historical data mining was proposed. First, an intelligent identification method of outliers of single-source data is proposed based on neural tangent kernel K-means (NTKKM) clustering. The original data is mapped to high-dimensional feature space using Neural Tangent Kernel, where the features of outliers are acquired by K-means clustering to realize the accurate identification of outliers. Second, an order-sensitive missing value imputation framework for multi-source data (OMSMVI) was proposed. The similarity graph of sources with missing data was constructed based on multidimensional similarity analysis, and the filling order decision was transformed into an optimization problem to realize the optimal filling order decision of missing values in multi-source data. Finally, a neighborhood-based imputation (NI) algorithm is proposed. Based on the traditional KNN filling algorithm, neighboring nodes of sources with missing data are flexibly selected to the achieve accurate correction of outliers. The case experiment was operated on actual power grid data, and the results show that the proposed clustering method can identify outliers more accurately, and the determined optimal imputation sequence has higher accuracy, which provide a feasible new idea for the identification and correction of outliers in the process of data preprocessing.Keywords
This publication has 17 references indexed in Scilit:
- Missing data in experiments: Challenges and solutions.Psychological Methods, 2022
- Object Weighting: A New Clustering Approach to Deal with Outliers and Cluster Overlap in Computational BiologyIEEE/ACM Transactions on Computational Biology and Bioinformatics, 2021
- A Fast Abnormal Data Cleaning Algorithm for Performance Evaluation of Wind TurbineIEEE Transactions on Instrumentation and Measurement, 2020
- Multiple Kernel k-Means Clustering by Selecting Representative KernelsIEEE Transactions on Neural Networks and Learning Systems, 2020
- A survey of data partitioning and sampling methods to support big data analysisBig Data Mining and Analytics, 2020
- Methodolgical Progress Note: Handling Missing Data in Clinical ResearchJournal of Hospital Medicine, 2019
- Enhanced Ensemble Clustering via Fast Propagation of Cluster-Wise SimilaritiesIEEE Transactions on Systems, Man, and Cybernetics: Systems, 2018
- The Kernel Conjugate Gradient AlgorithmsIEEE Transactions on Signal Processing, 2018
- Review on mining data from multiple data sourcesPattern Recognition Letters, 2018
- A Missing Data Approach to Data-Driven Filtering and ControlIEEE Transactions on Automatic Control, 2016