Empirical Case Studies in Attribute Noise Detection
Authors:
- Taghi M. KhoshgoftaarTKTaghi M. Khoshgoftaar
- Jason Van HulseJHJason Van Hulse
- 17 April 2009
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews)
- Vol. 39 (4) , 379-388
Abstract
The quality of data is an important issue in any domain-specific data mining and knowledge discovery initiative. The validity of solutions produced by data-driven algorithms can be diminished if the data being analyzed are of low quality. The quality of data is often realized in terms of data noise present in the given dataset and can include noisy attributes or labeling errors. Hence, tools for improving the quality of data are important to the data mining analyst. We present a comprehensive empirical investigation of our new and innovative technique for ranking attributes in a given dataset from most to least noisy. Upon identifying the noisy attributes, specific treatments can be applied depending on how the data are to be used. In a classification setting, for example, if the class label is determined to contain the most noise, processes to cleanse this important attribute may be undertaken. Independent variables or predictors that have a low correlation to the class attribute and appear noisy may be eliminated from the analysis. Several case studies using both real-world and synthetic datasets are presented in this study. The noise detection performance is evaluated by injecting noise into multiple attributes at different noise levels. The empirical results demonstrate conclusively that our technique provides a very accurate and useful ranking of noisy attributes in a given dataset.Keywords
This publication has 12 references indexed in Scilit:
- The pairwise attribute noise detection algorithmKnowledge and Information Systems, 2006
- Empirical case studies in attribute noise detection2005
- The necessity of assuring quality in software measurement data2004
- Class Noise vs. Attribute Noise: A Quantitative StudyArtificial Intelligence Review, 2004
- Improving automated land cover mapping by identifying and eliminating mislabeled observations from training data2002
- Identifying Mislabeled Training DataJournal of Artificial Intelligence Research, 1999
- Cleansing Data for Mining and Warehousing1999
- Classification of Fault-Prone Software Modules: Prior Probabilities, Costs, and Model EvaluationEmpirical Software Engineering, 1998
- Real-world Data is Dirty: Data Cleansing and The Merge/Purge ProblemData Mining and Knowledge Discovery, 1998
- Feature selection for classificationIntelligent Data Analysis: An International Journal, 1997