Massive data clustering by multi-scale psychological observations
Open Access
- 8 October 2021
- journal article
- research article
- Published by Oxford University Press (OUP) in National Science Review
- Vol. 9 (2) , nwab183
Abstract
Clustering is the discovery of latent group structure in data and is a fundamental problem in artificial intelligence, and a vital procedure in data-driven scientific research over all disciplines. Yet, existing methods have various limitations, especially weak cognitive interpretability and poor computational scalability, when it comes to clustering massive datasets that are increasingly available in all domains. Here, by simulating the multi-scale cognitive observation process of humans, we design a scalable algorithm to detect clusters hierarchically hidden in massive datasets. The observation scale changes, following the Weber–Fechner law to capture the gradually emerging meaningful grouping structure. We validated our approach in real datasets with up to a billion records and 2000 dimensions, including taxi trajectories, single-cell gene expressions, face images, computer logs and audios. Our approach outperformed popular methods in usability, efficiency, effectiveness and robustness across different domains.Keywords
Funding Information
- National Key Research and Development Program of China (2020YFA0713900)
- National Natural Science Foundation of China (U1811461, 11690011)
- Natural Sciences and Engineering Research Council of Canada (RGPIN-2016-05024)
- National Science Foundation (DMS-1662139, DMS-1712591)
This publication has 25 references indexed in Scilit:
- Clustering by fast search and find of density peaksScience, 2014
- Data clustering: 50 years beyond K-meansPattern Recognition Letters, 2010
- Signaling Motifs and Weber's LawMolecular Cell, 2009
- Fast unfolding of communities in large networksJournal of Statistical Mechanics: Theory and Experiment, 2008
- Clustering by Passing Messages Between Data PointsScience, 2007
- Survey of Clustering AlgorithmsIEEE Transactions on Neural Networks, 2005
- Mean shift: a robust approach toward feature space analysisIEEE Transactions on Pattern Analysis and Machine Intelligence, 2002
- Clustering by scale-space filteringIEEE Transactions on Pattern Analysis and Machine Intelligence, 2000
- Time required for Judgements of Numerical InequalityNature, 1967
- Hierarchical Grouping to Optimize an Objective FunctionJournal of the American Statistical Association, 1963