Paper Info
Title
Design and implementation of an analytical framework for interference aware job scheduling on Apache Spark platform
Abstract
Apache Spark is one of the recently popularized open-source platforms that is increasingly being used for large-scale data analytic applications. However, while performance prediction in such systems is important for efficient job scheduling and optimizing resource allocation, interference among multiple Apache Spark jobs running concurrently in a virtualized environment makes it extremely difficult, which is addressed in this paper. Towards that, first, we develop data-driven analytical models to estimate the effect of interference among multiple Apache Spark jobs on job execution time in virtualized cloud environments. Next, we present the design of an interference aware job scheduling algorithm leveraging the developed analytical framework. We evaluated the accuracy of our models using four real-life applications (e.g., Page rank, K-means, Logistic regression, and Word count) on a 6 node cluster while running up to four jobs concurrently. Our experimental results show that the scheduling algorithm reduces the average execution time of individual jobs and the total execution time significantly, and ranges between 47 and 26% for individual jobs and 2–13% for total execution time respectively.
Year
DOI
Venue
2019
10.1007/s10586-017-1466-3
Cluster Computing
Keywords
Field
DocType
Apache Spark, Job scheduling, Performance interference modeling, Execution time prediction
Spark (mathematics),Scheduling (computing),Computer science,Word count,Resource allocation,Job scheduler,Interference (wave propagation),Performance prediction,Distributed computing,Cloud computing
Journal
Volume
Issue
ISSN
22
SUPnan
1573-7543
Citations 
PageRank 
References 
2
0.39
26
Authors
4
Name
Order
Citations
PageRank
王克文159154.88
Mohammad Maifi Hasan Khan223322.04
Nhan Nguyen3466.33
Swapna S. Gokhale486077.93