dislib: Large Scale High Performance Machine Learning in Python
Authors:
- 1 September 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 96-105
Abstract
In recent years, machine learning has proven to be an extremely useful tool for extracting knowledge from data. This can be leveraged in numerous research areas, such as genomics, earth sciences, and astrophysics, to gain valuable insight. At the same time, Python has become one of the most popular programming languages among researchers due to its high productivity and rich ecosystem. Unfortunately, existing machine learning libraries for Python do not scale to large data sets, are hard to use by non-experts, and are difficult to set up in high performance computing clusters. These limitations have prevented scientists to exploit the full potential of machine learning in their research. In this paper, we present and evaluate dislib, a distributed machine learning library on top of PyCOMPSs programming model that addresses the issues of other existing libraries. In our evaluation, we show that dislib can be up to 9 times faster, and can process data sets up to 16 times larger than other popular distributed machine learning libraries, such as MLlib. In addition to this, we also show how dislib can be used to reduce the computation time of a real scientific application from 18 hours to 17 minutes.Keywords
This publication has 23 references indexed in Scilit:
- XGBoost2016
- PyCOMPSs: Parallel computational workflows in PythonThe International Journal of High Performance Computing Applications, 2016
- Spark deployment and performance evaluation on the MareNostrum supercomputer2015
- Machine learning: Trends, perspectives, and prospectsScience, 2015
- TheTycho-Gaiaastrometric solutionAstronomy & Astrophysics, 2015
- MLI: An API for Distributed Machine Learning2013
- Towards DaaS 2.0: Enriching Data Models2013
- The NumPy Array: A Structure for Efficient Numerical ComputationComputing in Science & Engineering, 2011
- MapReduceCommunications of the ACM, 2008
- A high-performance, portable implementation of the MPI message passing interface standardParallel Computing, 1996