Omnibenchmark: transparent, reproducible, extensible and standardized orchestration of solo and collaborative benchmarks

Authors:
  • IMIzaskun Mallona
    Izaskun Mallona
  • ALAlmut Luetge
    Almut Luetge
  • BCBen Carrillo
    Ben Carrillo
  • DIDaniel Incicau
    Daniel Incicau
  • RGReto Gerber
    Reto Gerber
  • AMAidan Meara
    Aidan Meara
  • ASAnthony Sonrel
    Anthony Sonrel
  • CSCharlotte Soneson
    Charlotte Soneson
  • MRMark D. Robinson
    Mark D. Robinson
Preprint
Publisher website
Google Scholar
Abstract
Benchmarking involves designing, running and disseminating rigorous performance assessments of methods, most often for data analysis and software tools, but the process can also be applied to experimental systems. Ideally, a benchmarking system is used to facilitate the benchmarking process by providing a structured entrypoint to design, coordinate, execute, and store standardized benchmarks. We describe a novel benchmarking system, Omnibenchmark, that facilitates benchmark formalization and execution in both solo and community efforts. Omnibenchmark provides a flexible benchmark plan syntax (i.e., a configuration YAML file), dynamic workflow generation based on Snakemake, S3-compatible storage handling, and reproducible software environments using environment modules, Apptainer or Conda. Such a setup provides an unprecedented flexibility such that existing benchmark designs can be forked and extended, run separately or collaboratively, giving versioned and standardized result outputs and therefore much-needed transparency to the analysis and interpretation of benchmark results. Tutorials and installation instructions are available from https://omnibenchmark.org.