An Empirical Study of Several Information Theoretic Based Feature Extraction Methods for Classifying High Dimensional Low Sample Size Data
Open Access
- 6 May 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 69157-69172
Abstract
A high dimensional low sample size (HDLSS) dataset typically contains many features but a limited number of samples. It is commonly found in domains such as microarray data and medical imaging. When sample size is small, the population probability density function (PDF) of a HDLSS dataset may not be well represented, causing difficulties of applying feature selection or feature extraction methods for HDLSS data classification. In this paper, we explore the possibility of designing feature selection and feature extraction methods for HDLSS data classification by making loose assumption on the underlying PDF of a HDLSS dataset. Specifically, we propose to leverage on Correlation Explanation (CorEx), a recent unsupervised probabilistic graphical model that assumes (hierarchical) hidden structure for generating subsets of features that are conditionally independent. We benchmark the proposed method against frequently cited Information Theory based feature extraction and feature selection methods, including Conditional Infomax Feature Extraction (CIFE), Maximum Relevance Minimum Redundancy (MRMR), Maximization of Mutual Information (MMI), Infomax Independent Component Analysis (Infomax ICA),and Kernel Entropy Component Analysis (KECA). The HDLSS datasets used in this study are Breast Cancer Dataset by Gravier et. al and West et. al, Colon Cancer dataset by Alon et. al., Leukemia Dataset by Golub et.al and the Gisette Dataset used by Guyon et. al. Experimental results demonstrate that the proposed method shows some improvement in classification performance over MMI, and Infomax ICA and is competitive with MRMR and CIFE.Keywords
Funding Information
- Fundamental Research Grant Scheme
- Ministry of Education of Malaysia (FRGS/1/2014/ICT07/UNIM/02/1)
This publication has 20 references indexed in Scilit:
- Deep quality-related feature extraction for soft sensing modeling: A deep learning approach with hybrid VW-SAENeurocomputing, 2020
- Comprehensive discovery of subsample gene expression components by information explanation: therapeutic implications in cancerBMC Medical Genomics, 2017
- A Survey on Feature SelectionProcedia Computer Science, 2016
- A survey on feature selection methodsComputers and Electrical Engineering, 2014
- A prognostic DNA signature for T1T2 node‐negative breast cancer patientsGenes, Chromosomes and Cancer, 2010
- Kernel Entropy Component AnalysisIEEE Transactions on Pattern Analysis and Machine Intelligence, 2010
- PCA consistency in high dimension, low sample size contextThe Annals of Statistics, 2009
- Maximization of Mutual Information for Supervised Linear Feature ExtractionIEEE Transactions on Neural Networks, 2007
- Interpretations of Rényi entropies and divergencesPhysica A: Statistical Mechanics and its Applications, 2006
- Selection of relevant features and examples in machine learningArtificial Intelligence, 1997