Name
Abstract | ||
|---|---|---|
Straggler tasks continue to be a major hurdle in achieving faster completion of data intensive applications running on modern data-processing frameworks. Existing straggler mitigation techniques are inefficient due to their reactive and replicative nature -- they rely on a wait-speculate-re-execute mechanism, thus leading to delayed straggler detection and inefficient resource utilization. Existing proactive techniques also over-utilize resources due to replication. Existing modeling-based approaches are hard to rely on for production-level adoption due to modeling errors. We present Wrangler, a system that proactively avoids situations that cause stragglers. Wrangler automatically learns to predict such situations using a statistical learning technique based on cluster resource utilization counters. Furthermore, Wrangler introduces a notion of a confidence measure with these predictions to overcome the modeling error problems; this confidence measure is then exploited to achieve a reliable task scheduling. In particular, by using these predictions to balance delay in task scheduling against the potential for idling of resources, Wrangler achieves a speed up in the overall job completion time. For production-level workloads from Facebook and Cloudera's customers, Wrangler improves the 99th percentile job completion time by up to 61% as compared to speculative execution, a widely used straggler mitigation technique. Moreover, Wrangler achieves this speed-up while significantly improving the resource consumption (by up to 55%). |
Year | DOI | Venue |
|---|---|---|
2014 | 10.1145/2670979.2671005 | SoCC |
Keywords | Field | DocType |
design,distributed systems,experimentation,measurement,performance,cloud computing,debugging | Resource consumption,Speculative execution,Scheduling (computing),Computer science,Real-time computing,Statistical learning,Speedup,Cloud computing,Debugging | Conference |
Citations | PageRank | References |
19 | 0.71 | 28 |
Authors | ||
3 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Neeraja J. Yadwadkar | 1 | 70 | 4.43 |
| Ganesh Ananthanarayanan | 2 | 1407 | 72.93 |
| Randy H. Katz | 3 | 16819 | 3018.89 |