Direction-Projection-Permutation for High-Dimensional Hypothesis Tests
- 2 April 2016
- journal article
- high dimensional-methods
- Published by Taylor & Francis in Journal of Computational and Graphical Statistics
- Vol. 25 (2) , 549-569
Abstract
High-dimensional low sample size (HDLSS) data are becoming increasingly common in statistical applications. When the data can be partitioned into two classes, a basic task is to construct a classifier that can assign objects to the correct class. Binary linear classifiers have been shown to be especially useful in HDLSS settings and preferable to more complicated classifiers because of their ease of interpretability. We propose a computational tool called direction-projection-permutation (DiProPerm), which rigorously assesses whether a binary linear classifier is detecting statistically significant differences between two high-dimensional distributions. The basic idea behind DiProPerm involves working directly with the one-dimensional projections of the data induced by binary linear classifier. Theoretical properties of DiProPerm are studied under the HDLSS asymptotic regime whereby dimension diverges to infinity while sample size remains fixed. We show that certain variations of DiProPerm are consistent and that consistency is a nontrivial property of tests in the HDLSS asymptotic regime. The practical utility of DiProPerm is demonstrated on HDLSS gene expression microarray datasets. Finally, an empirical power study is conducted comparing DiProPerm to several alternative two-sample HDLSS tests to understand the advantages and disadvantages of each method.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- A nonparametric two-sample test applicable to high dimensional dataJournal of Multivariate Analysis, 2014
- Exact and asymptotically robust permutation testsThe Annals of Statistics, 2013
- Two-Sample Covariance Matrix Testing and Support Recovery in High-Dimensional and Sparse SettingsJournal of the American Statistical Association, 2013
- A two-sample test for high-dimensional data with applications to gene-set testingThe Annals of Statistics, 2010
- Race, Breast Cancer Subtypes, and Survival in the Carolina Breast Cancer StudyJAMA, 2006
- Permutation Methods: A Basis for Exact InferenceStatistical Science, 2004
- On a new multivariate two-sample testJournal of Multivariate Analysis, 2004
- Permutation tests for equality of distributions in high-dimensional settingsBiometrika, 2002
- Sums of Functions of Nearest Neighbor Distances, Moment Bounds, Limit Theorems and a Goodness of Fit TestThe Annals of Probability, 1983
- Multivariate Generalizations of the Wald-Wolfowitz and Smirnov Two-Sample TestsThe Annals of Statistics, 1979