How Do Users Design Scientific Workflows? The Case of Snakemake and Nextflow

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Scientific workflows automate the analysis of large-scale scientific data, fostering the reuse of data processing operators as well as the reproducibility and traceability of analysis results. In exploratory research, however, workflows are continuously adapted, utilizing a wide range of tools and software libraries, to test scientific hypotheses. Script-based workflow engines cater to the required flexibility through direct integration of programming primitives, but lack abstractions for interactive exploration of the workflow design by a user during workflow execution. In this paper, we study the requirements for the design of interactive workflows through the lens of existing workflow collections. Specifically, we focus on two widely used script-based workflow engines: Snakemake and Nextflow. For Snakemake, we collected workflows from 1602 GitHub repositories that are listed in the Snakemake workflow catalog. For Nextflow, we adopted the nf-core collection with workflows from 94 GitHub repositories as the basis for our study. Using these collections, we present insights on common structures in the design of the workflows. Moreover, we report on results on the language features typically adopted in workflow specification, with a focus on the concepts that are of particular importance for interactive workflows.
Funding Information
  • Deutsche Forschungsgemeinschaft (414984028)

This publication has 16 references indexed in Scilit: