Rethinking Data-Intensive Science Using Scalable Analytics Systems
Open Access
- 27 May 2015
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 631-646
Abstract
"Next generation" data acquisition technologies are allowing scientists to collect exponentially more data at a lower cost. These trends are broadly impacting many scientific fields, including genomics, astronomy, and neuroscience. We can attack the problem caused by exponential data growth by applying horizontally scalable techniques from current analytics systems to accelerate scientific processing pipelines. In this paper, we describe ADAM, an example genomics pipeline that leverages the open-source Apache Spark and Parquet systems to achieve a 28x speedup over current genomics pipelines, while reducing cost by 63%. From building this system, we were able to distill a set of techniques for implementing scientific analyses efficiently using commodity "big data" systems. To demonstrate the generality of our architecture, we then implement a scalable astronomy image processing system which achieves a 2.8--8.9x improvement over the state-of-the-art MPI-based system.Keywords
Funding Information
- Lawrence Berkely National Laboratory (7076018)
- National Science Foundation (CCF-1139158)
- National Institutes of Health (HHSN261201400006C,1-U54HG007990-01)
- Defense Advanced Research Projects Agency (FA8750-12-2-0331)
This publication has 46 references indexed in Scilit:
- Spark SQL2015
- SpeedSeq: Ultra-fast personal genome analysis and interpretationbioRxiv, 2014
- Analyzing neural data at huge scaleNature Methods, 2014
- From FastQ Data to High‐Confidence Variant Calls: The Genome Analysis Toolkit Best Practices PipelineCurrent Protocols in Bioinformatics, 2013
- Hadoop GISProceedings of the VLDB Endowment, 2013
- Abstractions for genomicsCommunications of the ACM, 2013
- A framework for variation discovery and genotyping using next-generation DNA sequencing dataNature Genetics, 2011
- Overview of sciDB2010
- MapReduceCommunications of the ACM, 2008
- Integrating compression and execution in column-oriented database systems2006