Name
Abstract | ||
|---|---|---|
Resource selection and task placement for distributed execution poses conceptual and implementation difficulties. Although resource selection and task placement are at the core of many tools and workflow systems, the models and methods are underdeveloped. Consequently, partial and non-interoperable implementations proliferate. We address both the conceptual and implementation difficulties by experimentally characterizing diverse modalities of resource selection and task placement. We compare the architectures and capabilities of two systems: the AIMES middleware and Swift workflow scripting language and runtime. We integrate these systems to enable the distributed execution of Swift workflows on Pilot-Jobs managed by the AIMES middleware. Our experiments characterize and compare alternative execution strategies by measuring the time to completion of heterogeneous uncoupled workloads executed at diverse scale and on multiple resources. We measure the adverse effects of pilot fragmentation and early binding of tasks to resources and the benefits of backfill scheduling across pilots on multiple resources. We then use this insight to execute a multi-stage workflow across five production-grade resources. We discuss the importance and implications for other tools and workflow systems. |
Year | Venue | Field |
|---|---|---|
2016 | arXiv: Distributed, Parallel, and Cluster Computing | Modalities,Middleware,Workflow technology,Swift,Computer science,Scheduling (computing),Real-time computing,Implementation,Workflow,Scripting language,Distributed computing |
DocType | Volume | Citations |
Journal | abs/1605.09513 | 1 |
PageRank | References | Authors |
0.36 | 8 | 7 |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Matteo Turilli | 1 | 84 | 16.21 |
| Yadu Nand Babuji | 2 | 1 | 0.70 |
| André Merzky | 3 | 273 | 41.56 |
| Ming Tai Ha | 4 | 1 | 0.70 |
| Michael Wilde | 5 | 228 | 27.80 |
| Daniel S. Katz | 6 | 1496 | 121.04 |
| Shantenu Jha | 7 | 188 | 32.40 |