Parallel compression for large collections of genomes
- 4 June 2021
- journal article
- research article
- Published by Wiley in Concurrency and Computation: Practice and Experience
- Vol. 34 (2) , e6339
Abstract
Summary: With the development of genome sequencing technology, the cost of genome sequencing is continuously reducing, while the efficiency is increasing. Therefore, the amount of genomic data has been increasing exponentially, making the transmission and storage of genomic data an enormous challenge. Although many excellent genome compression algorithms have been proposed, an efficient compression algorithm for large collections of FASTA genomes, especially can be used in the distributed system of cloud computing, is still lacking. This article proposes two optimization schemes based on HRCM compression method. One is MtHRCM adopting multi‐thread parallel technology. The other is HadoopHRCM adopting distributed computing parallel technology. Experiments show that the schemes recognizably improve the compression speed of HRCM. Moreover, BSC algorithm instead of PPMD algorithm is used in the new schemes, the compression ratio is improved by 20% compared with HRCM. In addition, our proposed methods also perform well in robustness and scalability. The Java source codes of MtHRCM and HadoopHRCM can be freely downloaded from https://github.com/haicy/MtHRCM and https://github.com/haicy/HadoopHRCM.Keywords
Funding Information
- National Natural Science Foundation of China (61902194)
This publication has 33 references indexed in Scilit:
- Adaptive efficient compression of genomesAlgorithms for Molecular Biology, 2012
- Data Management Challenges in Next Generation SequencingDatenbank-Spektrum, 2012
- DELIMINATE—a fast and efficient method for loss-less compression of genomic sequencesBioinformatics, 2012
- Single-molecule sequencing of an individual human genomeNature Biotechnology, 2009
- The first Korean genome sequence and analysis: Full genome sequencing for a socio-ethnic groupGenome Research, 2009
- Data structures and compression algorithms for genomic sequence dataBioinformatics, 2009
- The 1001 Genomes Project for Arabidopsis thalianaGenome Biology, 2009
- Human genomes as email attachmentsBioinformatics, 2008
- The Diploid Genome Sequence of an Individual HumanPLOS Biology, 2007
- Arithmetic coding for data compressionCommunications of the ACM, 1987