Paper Info
Title
Scheduling mapreduce jobs in HPC clusters
Abstract
MapReduce (MR) has become a de facto standard for large-scale data analysis. Moreover, it has also attracted the attention of the HPC community due to its simplicity, efficiency and highly scalable parallel model. However, MR implementations present some issues that may complicate its execution in existing HPC clusters, specially concerning the job submission. While on MR there are no strict parameters required to submit a job, in a typical HPC cluster, users must specify the number of nodes and amount of time required to complete the job execution. This paper presents the MR Job Adaptor, a component to optimize the scheduling of MR jobs along with HPC jobs in an HPC cluster. Experiments performed using real-world HPC and MapReduce workloads have show that MR Job Adaptor can properly transform MR jobs to be scheduled in an HPC Cluster, minimizing the job turnaround time, and exploiting unused resources in the cluster.
Year
DOI
Venue
2012
10.1007/978-3-642-32820-6_19
international conference on parallel processing
Keywords
DocType
Volume
mapreduce job,mr implementation,mr job,job submission,hpc cluster,hpc job,hpc community,job execution,real-world hpc,typical hpc cluster,mr job adaptor
Conference
7484
ISSN
Citations 
PageRank 
0302-9743
4
0.51
References 
Authors
14
3
Name
Order
Citations
PageRank
Marcelo Veiga Neves1493.80
Tiago Ferreto2745.50
César A. F. De Rose3198287.05