Name
Abstract | ||
|---|---|---|
Map Reduce cluster is emerging as a solution of data-intensive scalable computing system. The open source implementation Hadoop has already been adopted for building clusters containing thousands of nodes. Such cloud infrastructure was used to processing many different jobs depending on different hardware resources, such as memory, CPU, Disk I/O and Network I/O, simultaneously. If the schedule policy does not consider the heterogeneity of running jobs' resource utilization types, resource contention may happen. In this paper, we analyze this multiple job parallelization problems in Map Reduce, and propose the multiple-job optimization (MJO) scheduler. Our scheduler detects job's resource utilization type on the fly and improves the hardware utilization by parallel different kinds of jobs. We give two scenarios which are “same plan” and “same job” to illustrate the multiple jobs' submission traces in Map Reduce clusters. Our experiments show that in these scenarios, MJO scheduler could save the make span by about 20%. |
Year | DOI | Venue |
|---|---|---|
2010 | 10.1109/SKG.2010.23 | SKG |
Keywords | Field | DocType |
public domain software,parallel processing,map reduce cluster,mapreduce,scheduling,resource utilization,schdule,hardware utilization,resource contention,data-intensive scalable computing system,multiple-job optimization scheduler,multiple job parallelization problems,cloud infrastructure,heterogeneous workloads,hadoop open source software,resource allocation,multiple-job optimization,multiple job,multiple job parallelization problem,mapreduce cluster,different hardware resource,mutiple job optimization,map reduce,different job,mjo scheduler,resource utilization type,parallel different kind,schedules,hardware,optimization,computational modeling | Cluster (physics),Scheduling (computing),Resource contention,Computer science,Schedule,Resource allocation,Job scheduler,Job queue,Operating system,Distributed computing,Cloud computing | Conference |
ISBN | Citations | PageRank |
978-0-7695-4189-1 | 10 | 0.55 |
References | Authors | |
9 | 8 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Weisong Hu | 1 | 62 | 5.76 |
| Chao Tian | 2 | 95 | 9.54 |
| Xiaowei Liu | 3 | 23 | 5.75 |
| Hongwei Qi | 4 | 20 | 4.00 |
| Li Zha | 5 | 10 | 0.55 |
| Huaming Liao | 6 | 57 | 5.09 |
| Yuezhuo Zhang | 7 | 12 | 1.59 |
| Jie Zhang | 8 | 841 | 185.41 |