A New Separation Index and Classification Techniques Based on Shannon Entropy
Open Access
- 22 September 2023
- journal article
- research article
- Published by Springer Nature in Methodology and Computing in Applied Probability
- Vol. 25 (4) , 1-24
Abstract
The purpose is to use Shannon entropy measures to develop classification techniques and an index which estimates the separation of the groups in a finite mixture model. These measures can be applied to machine learning techniques such as discriminant analysis, cluster analysis, exploratory data analysis, etc. If we know the number of groups and we have training samples from each group (supervised learning) the index is used to measure the separation of the groups. Here some entropy measures are used to classify new individuals in one of these groups. If we are not sure about the number of groups (unsupervised learning), the index can be used to determine the optimal number of groups from an entropy (information/uncertainty) criterion. It can also be used to determine the best variables in order to separate the groups. In all the cases we assume that we have absolutely continuous random variables and we use the Shannon entropy based on the probability density function. Theoretical, parametric and non-parametric techniques are proposed to get approximations of these entropy measures in practice. An application to gene selection in a colon cancer discrimination study with a lot of variables is provided as well.Keywords
Funding Information
- Universidad de Murcia
This publication has 14 references indexed in Scilit:
- Arbitrarily Tight Bounds on Differential Entropy of Gaussian MixturesIEEE Transactions on Information Theory, 2016
- On dynamic mutual information for bivariate lifetimesAdvances in Applied Probability, 2015
- Exploring correlations in gene expression microarray data for maximum predictive–minimum redundancy biomarker selection and classificationComputers in Biology and Medicine, 2013
- Properties of classical and quantum Jensen-Shannon divergencePhysical Review A, 2009
- Cumulative Residual Entropy: A New Measure of InformationIEEE Transactions on Information Theory, 2004
- Entropy-based measure of uncertainty in past lifetime distributionsJournal of Applied Probability, 2002
- Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arraysProceedings of the National Academy of Sciences, 1999
- An entropy criterion for assessing the number of clusters in a mixture modelJournal of Classification, 1996
- Possible generalization of Boltzmann-Gibbs statisticsJournal of Statistical Physics, 1988
- A Mathematical Theory of CommunicationBell System Technical Journal, 1948