Towards a gold standard for benchmarking gene set enrichment analysis
Preprint
Abstract
Background: Although gene set enrichment analysis has become an integral part of high-throughput gene expression data analysis, the assessment of enrichment methods remains rudimentary and ad hoc. In the absence of suitable gold standards, evaluations are commonly restricted to selected data sets and biological reasoning on the relevance of resulting enriched gene sets. However, this is typically incomplete and biased towards the goals of individual investigations. Results: We present a general framework for standardized and structured benchmarking of enrichment methods based on defined criteria for applicability, gene set prioritization, and detection of relevant processes. This framework incorporates a curated compendium of 75 expression data sets investigating 42 different human diseases. The compendium features microarray and RNA-seq measurements, and each dataset is associated with a precompiled GO/KEGG relevance ranking for the corresponding disease under investigation. We perform a comprehensive assessment of 10 major enrichment methods on the benchmark compendium, identifying significant differences in (i) runtime and applicability to RNA-seq data, (ii) fraction of enriched gene sets depending on the type of null hypothesis tested, and (iii) recovery of the a priori defined relevance rankings. Based on these findings, we make practical recommendations on (i) how methods originally developed for microarray data can efficiently be applied to RNA-seq data, (ii) how to interpret results depending on the type of gene set test conducted, and (iii) which methods are best suited to effectively prioritize gene sets with high relevance for the phenotype investigated. Conclusion: We carried out a systematic assessment of existing enrichment methods, and identified best performing methods, but also general shortcomings in how gene set analysis is currently conducted. We provide a directly executable benchmark system for straightforward assessment of additional enrichment methods. Availability: http://bioconductor.org/packages/GSEABenchmarkeRKeywords
All Related Versions
- Published version: Briefings in Bioinformatics, 22 (1), 545.
This publication has 56 references indexed in Scilit:
- The Cancer Genome Atlas Pan-Cancer analysis projectNature Genetics, 2013
- Measurement of mRNA abundance using RNA-seq data: RPKM measure is inconsistent among samplesTheory in Biosciences, 2012
- Differential gene and transcript expression analysis of RNA-seq experiments with TopHat and CufflinksNature Protocols, 2012
- Gene set enrichment analysis: performance evaluation and usage guidelinesBriefings in Bioinformatics, 2011
- From sets to graphs: towards a realistic enrichment analysis of transcriptomic systemsBioinformatics, 2011
- ROAST: rotation gene set tests for complex microarray experimentsBioinformatics, 2010
- edgeR: a Bioconductor package for differential expression analysis of digital gene expression dataBioinformatics, 2009
- Systematic and integrative analysis of large gene lists using DAVID bioinformatics resourcesNature Protocols, 2008
- Bioinformatics enrichment tools: paths toward the comprehensive functional analysis of large gene listsNucleic Acids Research, 2008
- Gene set enrichment analysis: A knowledge-based approach for interpreting genome-wide expression profilesProceedings of the National Academy of Sciences of the United States of America, 2005