Dask: Parallel Computation with Blocked algorithms and Task Scheduling
Dask: Параллельные вычисления с использованием блочных алгоритмов и планирования задач
2015-01-01
SCID: 54.1/twgrcpaa
Discuss with AI
DaskNumPyblocked algorithmsdynamic task schedulingout-of-core computation
Figures from the paper
Abstract (AI)
Dask enables parallel and out-of-core computation. We couple blocked algorithms with dynamic and memory aware task scheduling to achieve a parallel and out-of-core NumPy clone. We show how this extends the effective scale of modern hardware to larger datasets and discuss how these ideas can be more broadly applied to other parallel collections.
Key Findings
1
Dask combines blocked algorithms with dynamic, memory-aware task scheduling for parallel and out-of-core computation.
2
Dask extends the effective scale of modern hardware to datasets larger than available memory.
3
The paper argues that blocked algorithms and memory-aware task scheduling can generalize to other parallel collection types.
4
The system provides a parallel, out-of-core computational framework that functions as a NumPy clone.
Research Object
Dask's parallel and out-of-core computation system for blocked array algorithms
Research Subject
The scalability and execution efficiency achieved by coupling blocked algorithms with dynamic, memory-aware task scheduling
Publication Details
Publication Date
2015-01-01
Journal
Publisher
ISSN
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest