Higher criticism thresholding: Optimal feature selection when useful features are rare and weak

In important application fields today—genomics and proteomics are examples—selecting a small subset of useful features is crucial for success of Linear Classification Analysis. We study feature selection by thresholding of featureZ-scores and introduce a principle of threshold selection, based on the notion ofhigher criticism(HC). Fori= 1, 2, …,p, let π_idenote the two-sidedP-value associated with theith featureZ-score and π_(i)denote theith order statistic of the collection ofP-values. The HC threshold is the absoluteZ-score corresponding to theP-value maximizing the HC objective (i/p− π_(i))/i/p(1−i/p). We consider a rare/weak (RW) feature model, where the fraction of useful features is small and the useful features are each too weak to be of much use on their own. HC thresholding (HCT) has interesting behavior in this setting, with an intimate link between maximizing the HC objective and minimizing the error rate of the designed classifier, and very different behavior from popular threshold selection procedures such as false discovery rate thresholding (FDRT). In the most challenging RW settings, HCT uses an unconventionally low threshold; this keeps the missed-feature detection rate under better control than FDRT and yields a classifier with improved misclassification performance. Replacing cross-validated threshold selection in the popular Shrunken Centroid classifier with the computationally less expensive and simpler HCT reduces the variance of the selected threshold and the error rate of the constructed classifier. Results on standard real datasets and in asymptotic theory confirm the advantages of HCT.