Dask: Parallel Computation with Blocked algorithms and Task Scheduling
Top Cited Papers
Open Access
- 1 January 2015
- conference paper
- Published by NumFOCUS - Insight Software Consortium (ITK)
- p. 126-132
Abstract
Dask enables parallel and out-of-core computation. We couple blocked algorithms with dynamic and memory aware task scheduling to achieve a parallel and out-of-core NumPy clone. We show how this extends the effective scale of modern hardware to larger datasets and discuss how these ideas can be more broadly applied to other parallel collections.Keywords
This publication has 5 references indexed in Scilit:
- Compressed Nonnegative Matrix Factorization Is Fast and AccurateIEEE Transactions on Signal Processing, 2016
- The NumPy Array: A Structure for Efficient Numerical ComputationComputing in Science & Engineering, 2011
- Numerical linear algebra on emerging architectures: The PLASMA and MAGMA projectsJournal of Physics: Conference Series, 2009
- A class of parallel tiled linear algebra algorithms for multicore architecturesParallel Computing, 2009
- Python for Scientific ComputingComputing in Science & Engineering, 2007