Sijin He

Developing a robust and performant data analysis workflow that integrates all necessary components whilst still being able to scale over multiple compute nodes is a challenging task. We introduce a generic method based on the microservice architecture, where software tools are encapsulated as Docker containers that can be connected into scientific workflows and executed using the Kubernetes container orchestrator. We developed a Virtual Research Environment (VRE) which facilitates rapid integration of new tools and developing scalable and interoperable workflows for performing metabolomics data analysis. The environment can be launched on-demand on cloud resources and desktop computers. IT-expertise requirements on the user side are kept to a minimum, and workflows can be re-used effortlessly by any novice user. We validate our method in the field of metabolomics on two mass spectrometry, one nuclear magnetic resonance spectroscopy and one fluxomics study. We showed that the method scales dynamically with increasing availability of computational resources. We demonstrated that the method facilitates interoperability using integration of the major software suites resulting in a turn-key workflow encompassing all steps for mass-spectrometry-based metabolomics including preprocessing, statistics and identification. Microservices is a generic methodology that can serve any scientific discipline and opens up for new types of large-scale integrative science. The PhenoMeNal consortium maintains a web portal (https://portal.phenomenal-h2020.eu) providing a GUI for launching the Virtual Research Environment. The GitHub repository https://github.com/phnmnl/ hosts the source code of all projects. Supplementary data are available at Bioinformatics online.

Summary: Making reproducible, auditable and scalable data-processing analysis workflows is an important challenge in the field of bioinformatics. Recently, software containers and cloud computing introduced a novel solution to address these challenges. They simplify software installation, management and reproducibility by packaging tools and their dependencies. In this work we implemented a cloud provider agnostic and scalable container orchestration setup for the popular Galaxy workflow environment. This solution enables Galaxy to run on and offload jobs to most cloud providers (e.g. Amazon Web Services, Google Cloud or OpenStack, among others) through the Kubernetes container orchestrator. Availability: All code has been contributed to the Galaxy Project and is available (since Galaxy 17.05) at https://github.com/galaxyproject/ in the galaxy and galaxy-kubernetes repositories. https://public.phenomenal-h2020.eu/ is an example deployment. Suppl. Information: Supplementary Files are available online. Contact: pmoreno@ebi.ac.uk , European Molecular Biology Laboratory, EMBL-EBI, Wellcome Trust Genome Campus, Hinxton, Cambridge, CB10 1SD, UK, Tel: +44-1223-494267, Fax: +44-1223-484696.

Developing a robust and performant data analysis workflow that integrates all necessary components whilst still being able to scale over multiple compute nodes is a challenging task. We introduce a generic method based on the microservice architecture, where software tools are encapsulated as Docker containers that can be connected into scientific workflows and executed in parallel using the Kubernetes container orchestrator. The access point is a virtual research environment which can be launched on-demand on cloud resources and desktop computers. IT-expertise requirements on the user side are kept to a minimum, and established workflows can be re-used effortlessly by any novice user. We validate our method in the field of metabolomics on two mass spectrometry studies, one nuclear magnetic resonance spectroscopy study and one fluxomics study, showing that the method scales dynamically with increasing availability of computational resources. We achieved a complete integration of the major software suites resulting in the first turn-key workflow encompassing all steps for mass-spectrometry-based metabolomics including preprocessing, multivariate statistics, and metabolite identification. Microservices is a generic methodology that can serve any scientific discipline and opens up for new types of large-scale integrative science.

Motivation: TranSMART has a wide range of functionalities for translational research and a large user community, but it does not support imaging data. In this context, imaging data typically includes 2D or 3D sets of magnitude data and metadata information. Imaging data may summarise complex feature descriptions in a less biased fashion than user defined plain texts and numeric numbers. Imaging data also is contextualised by other data sets and may be analysed jointly with other data that can explain features or their variation. Results: Here we describe the tranSMART-XNAT Connector we have developed. This connector consists of components for data capture, organisation and analysis. Data capture is responsible for imaging capture either from PACS system or directly from an MRI scanner, or from raw data files. Data are organised in a similar fashion as tranSMART and are stored in a format that allows direct analysis within tranSMART. The connector enables selection and download of DICOM images and associated resources using subjects’ clinical phenotypic and genotypic criteria. Availability and Implementation: tranSMART-XNAT connector is written in Java/Groovy/Grails. It is maintained and available for download at https://github.com/sh107/transmart-xnat-connector.git Contact:sijin@ebi.ac.uk

Cloud applications have been gaining popularity in recent years for their flexibility in resource provisioning according to Web application demands. The Elastic Application Container (EAC) system is a technology that delivers a lightweight virtual resource unit for better resource efficiency and more scalable Web applications in the Cloud. It allows multiple application providers to concurrently run their Web applications on this technology without worrying the demand change of their Web applications. This is because the EAC system constantly monitors the resource usage of all hosting Web applications and automatically reacts to the resource usage change of Web applications (i.e. it automatically handles resource provisioning of the Web applications, such as scaling of the Web applications according to the demand). In the chapter, the authors firstly describe the architecture, its components of the EAC system, in order to give a brief overview of technologies involved in the system. They then present and explain resource-provisioning algorithms and techniques used in the EAC system for demand-driven Web applications. The resource-provisioning algorithms are presented, discussed, and evaluated so as to give readers a clear picture of resource-provisioning algorithms in the EAC system. Finally, the authors compare this EAC system technology with other Cloud technologies in terms of flexibility and resource efficiency.

High-throughput transcriptomic data generated by microarray experiments is the most abundant and frequently stored kind of data currently used in translational medicine studies. Although microarray data is supported in data warehouses such as tranSMART, when querying relational databases for hundreds of different patient gene expression records queries are slow due to poor performance. Non-relational data models, such as the key-value model implemented in NoSQL databases, hold promise to be more performant solutions. Our motivation is to improve the performance of the tranSMART data warehouse with a view to supporting Next Generation Sequencing data. In this paper we introduce a new data model better suited for high-dimensional data storage and querying, optimized for database scalability and performance. We have designed a key-value pair data model to support faster queries over large-scale microarray data and implemented the model using HBase, an implementation of Google's BigTable storage system. An experimental performance comparison was carried out against the traditional relational data model implemented in both MySQL Cluster and MongoDB, using a large publicly available transcriptomic data set taken from NCBI GEO concerning Multiple Myeloma. Our new key-value data model implemented on HBase exhibits an average 5.24-fold increase in high-dimensional biological data query performance compared to the relational model implemented on MySQL Cluster, and an average 6.47-fold increase on query performance on MongoDB. The performance evaluation found that the new key-value data model, in particular its implementation in HBase, outperforms the relational model currently implemented in tranSMART. We propose that NoSQL technology holds great promise for large-scale data management, in particular for high-dimensional biological data such as that demonstrated in the performance evaluation described in this paper. We aim to use this new data model as a basis for migrating tranSMART's implementation to a more scalable solution for Big Data.

Cloud applications have been gaining popularity in recent years for their flexibility in resource provisioning according to Web application demands. The Elastic Application Container (EAC) system is a technology that delivers a lightweight virtual resource unit for better resource efficiency and more scalable Web applications in the Cloud. It allows multiple application providers to concurrently run their Web applications on this technology without worrying the demand change of their Web applications. This is because the EAC system constantly monitors the resource usage of all hosting Web applications and automatically reacts to the resource usage change of Web applications (i.e. it automatically handles resource provisioning of the Web applications, such as scaling of the Web applications according to the demand). In the chapter, the authors firstly describe the architecture, its components of the EAC system, in order to give a brief overview of technologies involved in the system. They then present and explain resource-provisioning algorithms and techniques used in the EAC system for demand-driven Web applications. The resource-provisioning algorithms are presented, discussed, and evaluated so as to give readers a clear picture of resource-provisioning algorithms in the EAC system. Finally, the authors compare this EAC system technology with other Cloud technologies in terms of flexibility and resource efficiency.

We present a novel security monitoring framework for intrusion detection in IaaS cloud infrastructures. The framework uses statistical anomaly detection techniques over data monitored both inside and outside each Virtual Machine instance. We present the architecture of our monitoring framework and describe the implementation of the real-time monitors and detectors. We also describe how the framework is used in three different attack scenarios. For each of the three attack scenarios, we describe how the attack itself works and how it could be detected. We describe what data is monitored in our framework and how the detection is conducted using anomaly detection methods. We also present evaluation of the detection using synthetic and real data sets. Our experimental evaluation across all three scenarios shows that our tools perform well in practical situations and provide a promising direction for future research.

Resource provisioning based on virtual machine (VM) has been widely accepted and adopted in cloud computing environments. A key problem resulting from using static scheduling approaches for allocating VMs on different physical machines (PMs) is that resources tend to be not fully utilised. Although some existing cloud reconfiguration algorithms have been developed to address the problem, they normally result in high migration costs and low resource utilisation due to ignoring the multi-dimensional characteristics of VMs and PMs. In this paper we present and evaluate a new algorithm for improving resource utilisation for cloud providers. By using a multivariate probabilistic model, our algorithm selects suitable PMs for VM re-allocation which are then used to generate a reconfiguration plan. We also describe two heuristics metrics which can be used in the algorithm to capture the multi-dimensional characteristics of VMs and PMs. By combining these two heuristics metrics in our experiments, we observed that our approach improves the resource utilisation level by around 8% for cloud providers, such as IC Cloud, which accept user-defined VM configurations and 14% for providers, such as Amazon EC2, which only provide limited types of VM configurations.

Virtual machine (VM) based virtual infrastructure has been adopted widely in cloud computing environment for elastic resource provisioning. Performing resource management using VMs, however, is a heavyweight task. In practice, we have identified two scenarios where VM based resource management is less feasible and less resource-efficient. In this paper, we propose a lightweight resource management model that is called Elastic Application Container (EAC). EAC is a virtual resource unit for delivering better resource efficiency and more scalable cloud applications. We describe the EAC system architecture and components, and also present an algorithm for EAC resource provisioning. We also describe an implementation of the EAC-oriented platform to support multi-tenant cloud use. To evaluate our approach and implementation, we conducted experiments and collected performance data by comparing VM-based and EAC-based resource management with regards to their feasibility and resource-efficiency. The experiment results show that our proposed EAC-based resource management approach outperforms the VM-based approach in terms of feasibility and resource-efficiency.