Differentiable and Scalable Generative Adversarial Models for Data Imputation
- 7 July 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Knowledge and Data Engineering
- Vol. PP (10414347) , 1-13
Abstract
Data imputation has been extensively explored to solve the missing data problem. The dramatically increasing volume of incomplete data makes the imputation models computationally infeasible in many real-life applications. In this paper, we propose an effective scalable imputation system named SCIS to significantly speed up the training of the differentiable generative adversarial imputation models under accuracy-guarantees for large-scale incomplete data. SCIS consists of two modules, differentiable imputation modeling (DIM) and sample size estimation (SSE). DIM leverages a new masking Sinkhorn divergence function to make an arbitrary generative adversarial imputation model differentiable, while for such a differentiable imputation model, SSE can estimate an appropriate sample size to ensure the user-specified imputation accuracy of the final model. Moreover, SCIS can also accelerate the autoencoder based imputation models. Extensive experiments upon several real-life large-scale datasets demonstrate that, our proposed system can accelerate the generative adversarial model training by 6.23x. Using around 1.27% samples, SCIS yields competitive accuracy with the state-of-the-art imputation methods in much shorter computation time.Keywords
All Related Versions
This publication has 30 references indexed in Scilit:
- MIDA: Multiple Imputation Using Denoising Autoencoders2018
- HoloCleanProceedings of the VLDB Endowment, 2017
- XGBoost2016
- MissForest—non-parametric missing value imputation for mixed-type dataBioinformatics, 2011
- Multiple Imputation by Chained Equations (MICE): Implementation inStataJournal of Statistical Software, 2011
- AN EMPIRICAL COMPARISON OF TECHNIQUES FOR HANDLING INCOMPLETE DATA USING DECISION TREESApplied Artificial Intelligence, 2009
- A Novel Framework for Imputation of Missing Values in DatabasesIEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 2007
- Chapter 36 Large sample estimation and hypothesis testing1994
- An Introduction to Kernel and Nearest-Neighbor Nonparametric RegressionThe American Statistician, 1992
- A Relationship Between Arbitrary Positive Matrices and Doubly Stochastic MatricesThe Annals of Mathematical Statistics, 1964