Name
Title | ||
|---|---|---|
Turbine: a distributed-memory dataflow engine for extreme-scale many-task applications |
Abstract | ||
|---|---|---|
Efficiently utilizing the rapidly increasing concurrency of multi-petaflop computing systems is a significant programming challenge. One approach is to structure applications with an upper layer of many loosely-coupled coarse-grained tasks, each comprising a tightly-coupled parallel function or program. "Many-task" programming models such as functional parallel dataflow may be used at the upper layer to generate massive numbers of tasks, each of which generates significant tighly-coupled parallelism at the lower level via multithreading, message passing, and/or partitioned global address spaces. At large scales, however, the management of task distribution, data dependencies, and inter-task data movement is a significant performance challenge. In this work, we describe Turbine, a new highly scalable and distributed many-task dataflow engine. Turbine executes a generalized many-task intermediate representation with automated self-distribution, and is scalable to multi-petaflop infrastructures. We present here the architecture of Turbine and its performance on highly concurrent systems. |
Year | DOI | Venue |
|---|---|---|
2012 | 10.1145/2443416.2443421 | SWEET@SIGMOD |
Keywords | Field | DocType |
inter-task data movement,significant tighly-coupled parallelism,functional parallel dataflow,multi-petaflop computing system,distributed-memory dataflow engine,significant performance challenge,extreme-scale many-task application,upper layer,generalized many-task intermediate representation,data dependency,many-task dataflow engine,significant programming challenge,swift,concurrency,dataflow,mpi | Multithreading,Signal programming,Dataflow architecture,Programming paradigm,Concurrency,Computer science,Parallel computing,Distributed memory,Dataflow,Message passing,Database,Distributed computing | Conference |
Citations | PageRank | References |
12 | 0.62 | 24 |
Authors | ||
7 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Justin M. Wozniak | 1 | 464 | 35.32 |
| Timothy G. Armstrong | 2 | 373 | 21.73 |
| Ketan Maheshwari | 3 | 163 | 13.70 |
| Ewing L. Lusk | 4 | 3080 | 472.23 |
| Daniel S. Katz | 5 | 1496 | 121.04 |
| Michael Wilde | 6 | 12 | 0.62 |
| Foster Ian | 7 | 22938 | 2663.24 |