Name
Abstract | ||
|---|---|---|
Virtualization promised to dramatically increase server utilization levels, yet many data centers are still only lightly loaded. In some ways, big data applications are an ideal fit for using this residual capacity to perform meaningful work, but the high level of interference between interactive and batch processing workloads currently prevents this from being a practical solution in virtualized environments. Further, the variable nature of spare capacity may make it difficult to meet big data application deadlines. In this work we propose two schedulers: one in the virtualization layer designed to minimize interference on high priority interactive services, and one in the Hadoop framework that helps batch processing jobs meet their own performance deadlines. Our approach uses performance models to match Hadoop tasks to the servers that will benefit them the most, and deadline-aware scheduling to effectively order incoming jobs. The combination of these schedulers allows data center administrators to safely mix resource intensive Hadoop jobs with latency sensitive web applications, and still achieve predictable performance for both. We have implemented our system using Xen and Hadoop, and our evaluation shows that our schedulers allow a mixed cluster to reduce web response times by more than ten fold, while meeting more Hadoop deadlines and lowering total task execution times by 6.5%. |
Year | DOI | Venue |
|---|---|---|
2014 | 10.1109/CCGrid.2014.101 | CCGrid |
Keywords | Field | DocType |
batch processing jobs,interference minimization,virtualization,interactive processing workloads,scheduling,interference aware scheduling,computer centres,big data application deadlines,mimp,task execution times,map reduce,hadoop deadlines,server utilization levels,hadoop virtual machines,virtualization layer,virtual machines,latency sensitive web applications,deadline-aware scheduling,file servers,resource intensive hadoop jobs,data centers,virtualisation,internet,xen,web response times,performance deadlines,batch processing (computers),hadoop framework,deadlines,big data,high priority interactive services,interference,batch processing workloads,servers | Virtualization,Virtual machine,Computer science,Scheduling (computing),Server,Real-time computing,Batch processing,Web application,Big data,Data center,Operating system,Distributed computing | Conference |
ISSN | Citations | PageRank |
2376-4414 | 27 | 0.78 |
References | Authors | |
21 | 4 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Wei Zhang | 1 | 221 | 13.10 |
| Sundaresan Rajasekaran | 2 | 46 | 2.85 |
| Timothy Wood | 3 | 349 | 27.52 |
| Mingfa Zhu | 4 | 71 | 10.35 |