A Four-Phase Machine Learning Framework for Clustering and Feature Evaluation in High-Dimensional Observational Datasets
Authors:
- Nils KatzkeNKNils Katzke
- David Rößler-Von SaßDSDavid Rößler-Von Saß
- Christian HeinCHChristian Hein
- Natalia KliewerNKNatalia Kliewer
Preprint
- 4 August 2026
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
Unsupervised clustering in infrastructure, mobility, and energy research is highly sensitive to methodological choices regarding feature representation and algorithmic parameterization. This paper presents a reproducible, four-phase workflow designed for the systematic evaluation and comparison of clustering configurations in high-dimensional observational datasets. Rather than relying on a single unverified configuration, the framework combines automated dimensionality-reduction techniques, complementary clustering paradigms, quantitative validation via internal indices, and qualitative robustness analyses to verify that the identified structures reflect genuine underlying patterns. This approach effectively balances statistical quality, structural stability, and domain-specific interpretability. The core methodological highlights of this proposed framework are that it:- Provides a standardized, reproducible workflow to systematically contrast alternative feature-engineering pipelines and clustering algorithms rather than treating data preparation as an isolated preprocessing step.- Mitigates the curse of dimensionality in high-dimensional observational data by balancing statistical efficiency with information retention and cluster separability.- Integrates quantitative internal validity assessment with qualitative robustness criteria, such as feature selection consistency and cluster-size distribution analysis, ensuring practically useful and stable typologies.Keywords
This publication has 20 references indexed in Scilit:
- Machine Learning: Algorithms, Real-World Applications and Research DirectionsSN Computer Science, 2021
- A review of clustering techniques and developmentsNeurocomputing, 2017
- A survey on feature selection methodsComputers and Electrical Engineering, 2014
- Data clustering: 50 years beyond K-meansPattern Recognition Letters, 2010
- A simple and fast algorithm for K-medoids clusteringExpert Systems with Applications, 2009
- On Clustering Validation TechniquesJournal of Intelligent Information Systems, 2001
- Wrappers for feature subset selectionArtificial Intelligence, 1997
- Silhouettes: A graphical aid to the interpretation and validation of cluster analysisJournal of Computational and Applied Mathematics, 1987
- A dendrite method for cluster analysisCommunications in Statistics, 1974
- Hierarchical Grouping to Optimize an Objective FunctionJournal of the American Statistical Association, 1963