Paper Info
Title
Natjam: design and evaluation of eviction policies for supporting priorities and deadlines in mapreduce clusters
Abstract
This paper presents Natjam, a system that supports arbitrary job priorities, hard real-time scheduling, and efficient preemption for Mapreduce clusters that are resource-constrained. Our contributions include: i) exploration and evaluation of smart eviction policies for jobs and for tasks, based on resource usage, task runtime, and job deadlines; and ii) a work-conserving task preemption mechanism for Mapreduce. We incorporated Natjam into the Hadoop YARN scheduler framework (in Hadoop 0.23). We present experiments from deployments on a test cluster, Emulab and a Yahoo! Inc. commercial cluster, using both synthetic workloads as well as Hadoop cluster traces from Yahoo!. Our results reveal that Natjam incurs overheads as low as 7%, and is preferable to existing approaches.
Year
DOI
Venue
2013
10.1145/2523616.2523624
SoCC
Keywords
Field
DocType
task runtime,inc. commercial cluster,eviction policy,natjam incurs overhead,mapreduce cluster,hadoop yarn scheduler framework,arbitrary job priority,efficient preemption,test cluster,job deadline,hadoop cluster trace,scheduling
Cluster (physics),Preemption,Yarn,Computer science,Scheduling (computing),Real-time computing,Eviction,Operating system,Overhead (business),Distributed computing
Conference
Citations 
PageRank 
References 
34
1.06
42
Authors
7
Name
Order
Citations
PageRank
Brian Cho119915.57
Muntasir Raihan Rahman245521.21
Tej Chajed3895.41
Indranil Gupta41837143.92
Cristina Abad5866.51
Nathan Roberts6803.58
Philbert Lin7341.06