Dask: Parallel Computation with Blocked algorithms and Task Scheduling

Dask: Параллельные вычисления с использованием блочных алгоритмов и планирования задач
Matthew Rocklin
2015-01-01

DaskNumPyblocked algorithmsdynamic task schedulingout-of-core computation
Dask enables parallel and out-of-core computation.We couple blocked algorithms with dynamic and memory aware task scheduling to achieve a parallel and out-of-core NumPy clone.We show how this extends the effective scale of modern hardware to larger datasets and discuss how these ideas can be more broadly applied to other parallel collections.
1
Dask combines blocked algorithms with dynamic, memory-aware task scheduling for parallel and out-of-core computation.
2
Dask extends the effective scale of modern hardware to datasets larger than available memory.
3
The paper argues that blocked algorithms and memory-aware task scheduling can generalize to other parallel collection types.
4
The system provides a parallel, out-of-core computational framework that functions as a NumPy clone.

Dask's parallel and out-of-core computation system for blocked array algorithms

The scalability and execution efficiency achieved by coupling blocked algorithms with dynamic, memory-aware task scheduling

Publication Details
Publication Date
2015-01-01
Journal
Publisher
ISSN
Cited by
890
Access Type
Author Information
Authors
Matthew Rocklin
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%