Rapid development of cloud-native intelligent data pipelines for scientific data streams using the HASTE Toolkit
Authors:
Open Access
- 19 March 2021
- journal article
- research article
- Published by Oxford University Press (OUP) in GigaScience
- Vol. 10 (3)
Abstract
Background: Large streamed datasets, characteristic of life science applications, are often resource-intensive to process, transport and store. We propose a pipeline model, a design pattern for scientific pipelines, where an incoming stream of scientific data is organized into a tiered or ordered “data hierarchy". We introduce the HASTE Toolkit, a proof-of-concept cloud-native software toolkit based on this pipeline model, to partition and prioritize data streams to optimize use of limited computing resources. Findings: In our pipeline model, an “interestingness function” assigns an interestingness score to data objects in the stream, inducing a data hierarchy. From this score, a “policy” guides decisions on how to prioritize computational resource use for a given object. The HASTE Toolkit is a collection of tools to adopt this approach. We evaluate with 2 microscopy imaging case studies. The first is a high content screening experiment, where images are analyzed in an on-premise container cloud to prioritize storage and subsequent computation. The second considers edge processing of images for upload into the public cloud for real-time control of a transmission electron microscope. Conclusions: Through our evaluation, we created smart data pipelines capable of effective use of storage, compute, and network resources, enabling more efficient data-intensive experiments. We note a beneficial separation between scientific concerns of data priority, and the implementation of this behaviour for different resources in different deployment contexts. The toolkit allows intelligent prioritization to be `bolted on' to new and existing systems – and is intended for use with a range of technologies in different deployment scenarios.Keywords
All Related Versions
Funding Information
- Sjögren's Syndrome Foundation (BD15-0008)
This publication has 23 references indexed in Scilit:
- htsget: a protocol for securely streaming genomic dataBioinformatics, 2018
- ScipionCloud: An integrative and interactive gateway for large scale cryo electron microscopy image processing on commercial and academic cloudsJournal of Structural Biology, 2017
- The imaging tsunami: Computational opportunities and challengesCurrent Opinion in Systems Biology, 2017
- Robust High-dimensional Bioinformatics Data Streams Mining by ODR-ioVFDTScientific Reports, 2017
- Critical analysis of Big Data challenges and analytical methodsJournal of Business Research, 2017
- Real-time processing of proteomics data: The internet of things and the connected laboratory2016
- Scipion: A software framework toward integration, reproducibility and validation in 3D electron microscopyJournal of Structural Biology, 2016
- Big Data: Astronomical or Genomical?PLOS Biology, 2015
- Streaming visualisation of quantitative mass spectrometry data based on a novel raw signal decomposition methodProteomics, 2015
- Metabolomic data streaming for biology-dependent data acquisitionNature Biotechnology, 2014