Dear-OMG: An Omics-General Compression Method for Genomics, Proteomics and Metabolomics Data

Authors:
Publisher website
Google Scholar
Abstract
As high-throughput omics technologies continue to advance, researchers are facing the challenge of a rapid surge in proteomics, metabolomics, and genomics data. This growth not only necessitates more disk space and network bandwidth, but also complicates data sharing and subsequent analysis. To address this challenge and enhance the analytical efficiency of downstream software, we propose Dear-OMG, a unified, compact, flexible, and high-performance metadata storage solution. Dear-OMG introduces a novel file storage structure and utilizes the Elias-Fano encoding algorithm to compress and store proteomics, genomics, and metabolomics metadata into the unified OMG format. The OMG format not only demonstrates remarkably high compression and decompression speeds, but also enables parallel random access to any data block. Test results reveal that, compared to the commonly used proteomics formats of mzXML and mzML, the OMG format achieves an 80% reduction in storage space, a 90% decrease in conversion time, and approximately a tenfold speed improvement with support for parallel random access. Dear-OMG is freely available at https://github.com/jianweishuai/Dear-OMG. Graphical Abstract In omics data, proteomics and metabolomics apply high-throughput MS techniques to acquire mass spectral data for analyzing biological samples. These three-dimensional or four-dimensional mass spectral data are typically stored in vendor-specific formats (such as .d, .wiff, and .raw formats) or community formats (such as mzML, mzXML, imzML, and mzMLb formats). Single-cell genomics, on the other hand, studies gene expression through single-cell sequencing technology, with sequencing data stored in FASTQ files containing identifiers, gene sequences, and quality scores. Dear-OMG converts various omics storage formats into the unified OMG format. It offers three optional storage modes and two reading modes. Dear-OMG boasts several advantages, including minimal volume, fast encoding and decoding speeds, and support for random access and short conversion times.
Funding Information
  • Ministry of Science and Technology of the People's Republic of China (STI2030-Major Projects2021ZD0201900)
  • National Natural Science Foundation of China (12090052)
  • National Natural Science Foundation of China (12375334)
  • National Natural Science Foundation of China (U24A2014)
  • Natural Science Foundation of Fujian Province of China (2023J05002)

This publication has 46 references indexed in Scilit: