Data quality challenges with missing values and mixed types in joint sequence analysis
Authors:
- 1 December 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2620-2627
Abstract
The goal of this paper is to investigate the impact of missing values in categorical time series sequences on common data analysis tasks. Being able to more effectively identify patterns in socio-demographic longitudinal data is an important component in a number of social science settings. However, performing fundamental analytical operations, such as clustering for grouping these data based on similarity patterns, is challenging due to the categorical and multi-dimensional nature of the data, and their corruption by missing and inconsistent values. To study these data quality issues, we employ longitudinal sequence data representations, a similarity measure designed for categorical and longitudinal data, together with state-of-the art clustering methodologies reliant on hierarchical algorithms. The key to quantifying the similarity and difference among data records is a distance metric. Given the categorical nature of our data, we employ an “edit” type distance using Optimal Matching (OM). Because each data record has multiple variables of different types, we investigate the impact of mixing these variables in a single similarity measure. Between variables with binary values and those with multiple nominal values, we find that the ability to overcome missing data problems is harder in the nominal domain versus the binary domain. Additionally, artificial clusters introduced by the alignment of leading missing values can be resolved by tuning the missing value substitution cost parameter.Keywords
This publication has 24 references indexed in Scilit:
- Joint Sequence AnalysisSociological Methods & Research, 2016
- Analyzing and Visualizing State Sequences inRwithTraMineRJournal of Statistical Software, 2011
- 1. Multichannel Sequence Analysis Applied to Social Science DataSociological Methodology, 2010
- New Life for Old Ideas: The "Second Wave" of Sequence Analysis Bringing the "Course" Back Into the Life CourseSociological Methods & Research, 2010
- Holistic Trajectories: a Study of Combined Employment, Housing and Family Careers by using Multiple-Sequence AnalysisJournal of the Royal Statistical Society Series A: Statistics in Society, 2006
- Policy and sociologyThe British Journal of Sociology, 2004
- Sociology and political arithmetic: some principles of a new policy science1The British Journal of Sociology, 2004
- Making social science useful1The British Journal of Sociology, 2004
- Partitioning Around Medoids (Program PAM)1990
- The String-to-String Correction ProblemJournal of the ACM, 1974