Paper Info
Title
A Constraint Programming Based Hadoop Scheduler for Handling MapReduce Jobs with Deadlines on Clouds
Abstract
A novel MapReduce constraint programming based matchmaking and scheduling algorithm (MRCP) that can handle MapReduce jobs with deadlines and achieve high system performance is devised. The MRCP algorithm is incorporated into Hadoop, which is a widely used open source implementation of the MapReduce programming model, as a new scheduler called the CP-Scheduler. This paper originates from the collaborative research with our industrial partner concerning the engineering of resource management middleware for high performance. It describes our experiences and the challenges that we encountered in designing and implementing the prototype CP-based Hadoop scheduler. A detailed performance evaluation of the CP-Scheduler is conducted on Amazon EC2 to determine the CP-Scheduler's effectiveness as well as to obtain insights into system behaviour and performance. In addition, the CP-Scheduler's performance is also compared with an earliest deadline first (EDF) Hadoop scheduler, which is implemented by extending Hadoop's default FIFO scheduler. The experimental results demonstrate the effectiveness of the CP-Scheduler's ability to handle an open stream of MapReduce jobs with deadlines in a Hadoop cluster.
Year
DOI
Venue
2015
10.1145/2668930.2688058
ICPE
Keywords
DocType
Citations 
constraint programming.,distributed systems,mapreduce with deadlines,modeling techniques,resource management on clouds,performance attributes,hadoop scheduler,constraint programming
Conference
2
PageRank 
References 
Authors
0.37
10
3
Name
Order
Citations
PageRank
Norman Lim1274.10
Shikharesh Majumdar243575.95
Peter Ashwood-Smith3596.26