Enabling reproducible and reusable genetic demultiplexing benchmarking with Nextflow and Apptainer
Preprint
Abstract
Reproducibility, reusability, and portability were identified as key requirements for analysis pipelines in genomics. A previous study identified that less than 4% of notebooks used for biomedical research were fully reproducible. Reusable and portable software are required to ensure compatibility with federated and cloud computing. The field requires computational workflows that meet high standards for reproducibility, reusability, and portability to allow benchmarking to be repeated and updated periodically. This is of particular importance in areas where new methods are frequently released, such as demultiplexing scRNA-seq, a critical early step in most scRNA-seq analysis pipelines. To address this, we developed demux_bench for benchmarking genetic demultiplexing methods in single-cell RNA sequencing, which meets the gold standard for reproducibility of computational workflows, and incorporates best practices for reusability and portability. We used workflow manager Nextflow to enable the simulation and testing of various benchmarking scenarios and methods in parallel from a single pipeline execution. Different experimental configurations can be simulated, including % doublets and class size imbalance. The pipeline includes genotype-free methods, Vireo and souporcell, and additional existing or novel methods can be added modularly. Demux_bench is configurable to reproduce specific analyses and generalisable to address new research questions. Software dependencies are handled through containers via Apptainer, allowing portability to different compute environments and avoiding the need for manual installation of software. Demux_bench is available on WorkflowHub ( https://workflowhub.eu/workflows/1769 ), and can also be run on Galaxy and other platforms through RO-Crates. Demux_bench facilitates gold standard benchmarking for genetic demultiplexing through reproducibility, reusability, scalability, and portability.Keywords
This publication has 26 references indexed in Scilit:
- Against the “one method fits all data sets” philosophy for comparison studies in methodological researchBiometrical Journal, 2022
- Reproducible, scalable, and shareable analysis pipelines with bioinformatics workflow managersNature Methods, 2021
- Reproducibility standards for machine learning in the life sciencesNature Methods, 2021
- Souporcell: robust clustering of single-cell RNA-seq data by genotype without reference genotypesNature Methods, 2020
- Systematic benchmarking of omics computational toolsNature Communications, 2019
- Multiplexed droplet single-cell RNA-sequencing using natural genetic variationNature Biotechnology, 2017
- Nextflow enables reproducible computational workflowsNature Biotechnology, 2017
- iCOBRA: open, reproducible, standardized and live method benchmarkingNature Methods, 2016
- Orchestrating high-throughput genomic analysis with BioconductorNature Methods, 2015
- Repeatability of published microarray gene expression analysesNature Genetics, 2009