The Standard Deviation Score: a novel similarity metric for data analysis
Author:
Open Access
- 7 March 2025
- journal article
- research article
- Published by Springer Nature in Journal of Big Data
- Vol. 12 (1) , 1-42
Abstract
The ability to measure similarity or distance between data points is critical for various analytical tasks, including classification, clustering, and anomaly detection. However, traditional distance metrics such as Euclidean, Manhattan, and Hamming often struggle with mixed data types, varying attribute scales, and noise, limiting their robustness in diverse datasets. This paper introduces the Standard Deviation Score (SD-score), a novel similarity metric designed to address these challenges. By transforming traditional distance values into standard deviation units relative to a target point, the SD-score enables robust and interpretable similarity assessments. Extensive experimental evaluations demonstrate that the SD-score consistently outperforms conventional metrics in accuracy, precision, recall, and F-score within the k-Nearest Neighbors classification framework. Also, a comprehensive evaluation of the SD-score’s performance across Gaussian, skewed, and multimodal distributions showed promising results in the cluster coherence experiment, in which the Silhouette score was measured through the K-means clustering algorithm, emphasizing its adaptability to real-world data complexities. Additionally, the experiments detail improved handling of mixed numerical, ordinal, and categorical data types through a unified framework. The proposed metric incorporates inherent normalization mechanisms, reducing sensitivity to outliers and ensuring consistency across varying data scales and distributions, making it a versatile tool for real-world applications. This advancement in similarity measurement paves the way for more accurate and efficient data analysis across multiple domains.Keywords
Funding Information
- Cairo University
This publication has 30 references indexed in Scilit:
- K-means clustering algorithms: A comprehensive review, variants analysis, and advances in the era of big dataInformation Sciences, 2022
- Statistics for Data Scientists2022
- Multiclass classification of dry beans using computer vision and machine learning techniquesComputers and Electronics in Agriculture, 2020
- Application of self-organizing map to failure modes and effects analysis methodologyNeurocomputing, 2017
- Distance and Similarity Measures2017
- Locally centred Mahalanobis distance: A new distance measure with salient features towards outlier detectionAnalytica Chimica Acta, 2013
- Application support issues2010
- k-Nearest Neighbor Classification2009
- Molecular similarity analysis in virtual screening: foundations, limitations and novel approachesDrug Discovery Today, 2007
- Distance-based outliers: algorithms and applicationsThe VLDB Journal, 2000