Multivariate classification of neuroimaging data with nested subclasses: Biased accuracy and implications for hypothesis testing
Open Access
- 27 September 2018
- journal article
- research article
- Published by Public Library of Science (PLoS) in PLOS Computational Biology
- Vol. 14 (9) , e1006486
Abstract
Biological data sets are typically characterized by high dimensionality and low effect sizes. A powerful method for detecting systematic differences between experimental conditions in such multivariate data sets is multivariate pattern analysis (MVPA), particularly pattern classification. However, in virtually all applications, data from the classes that correspond to the conditions of interest are not homogeneous but contain subclasses. Such subclasses can for example arise from individual subjects that contribute multiple data points, or from correlations of items within classes. We show here that in multivariate data that have subclasses nested within its class structure, these subclasses introduce systematic information that improves classifiability beyond what is expected by the size of the class difference. We analytically prove that this subclass bias systematically inflates correct classification rates (CCRs) of linear classifiers depending on the number of subclasses as well as on the portion of variance induced by the subclasses. In simulations, we demonstrate that subclass bias is highest when between-class effect size is low and subclass variance high. This bias can be reduced by increasing the total number of subclasses. However, we can account for the subclass bias by using permutation tests that explicitly consider the subclass structure of the data. We illustrate our result in several experiments that recorded human EEG activity, demonstrating that parametric statistical tests as well as typical trial-wise permutation fail to determine significance of classification outcomes correctly. When data are analyzed using multivariate pattern classification, any systematic similarities between subsets of trials (e.g. shared physical properties among a subgroup of stimuli, trials belonging to the same session or subject, etc.) form distinct nested subclasses within each class. Pattern classification is sensitive to this kind of structure in the data and uses such groupings to increase classification accuracies even when data from both conditions are sampled from the same distribution, i.e. the null hypothesis is true. Here, we show that the bias is higher for larger subclass variances and that it is directly related to the number of subclasses and the intraclass correlation (ICC). Because the increased classification accuracy in such data sets is not based on class differences, the null distribution should be adjusted to account for this type of bias. To do so, we propose to use blocked permutation testing on subclass levels and show that it can confine the false positive rate to the predefined α-levels.Keywords
Funding Information
- Deutsche Forschungsgemeinschaft (GA730/3-1)
- Bundesministerium für Bildung und Forschung (BMBF) (01EO0901)
- Bundesministerium für Bildung und Forschung (BMBF) (01GQ1004A)
This publication has 24 references indexed in Scilit:
- Coping with confounds in multivoxel pattern analysis: What should we do about reaction time differences? A comment on Todd, Nystrom & Cohen 2013NeuroImage, 2014
- A solution to dependency: using multilevel analysis to accommodate nested dataNature Neuroscience, 2014
- Confounds in multivariate pattern analysis: Theory and rule representation case studyNeuroImage, 2013
- Statistical inference and multiple testing correction in classification-based multi-voxel pattern analysis (MVPA): Random permutations and cluster size controlNeuroImage, 2013
- A Study of Clustered Data and Approaches to Its AnalysisThe Journal of Neuroscience, 2010
- The problem of pseudoreplication in neuroscientific studies: is it affecting your analysis?BMC Neuroscience, 2010
- High-dimensional classification using features annealed independence rulesThe Annals of Statistics, 2008
- Beyond mind-reading: multi-voxel pattern analysis of fMRI dataTrends in Cognitive Sciences, 2006
- EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysisJournal of Neuroscience Methods, 2004
- Permutation tests for multi-factorial analysis of varianceJournal of Statistical Computation and Simulation, 2003