Paper Info
Title
Modeling Performance of Hadoop Applications: A Journey from Queueing Networks to Stochastic Well Formed Nets.
Abstract
Nowadays, many enterprises commit to the extraction of actionable knowledge from huge datasets as part of their core business activities. Applications belong to very different domains such as fraud detection or one-to-one marketing, and encompass business analytics and support to decision making in both private and public sectors. In these scenarios, a central place is held by the MapReduce framework and in particular its open source implementation, Apache Hadoop. In such environments, new challenges arise in the area of jobs performance prediction, with the needs to provide Service Level Agreement guarantees to the end-user and to avoid waste of computational resources. In this paper we provide performance analysis models to estimate MapReduce job execution times in Hadoop clusters governed by the YARN Capacity Scheduler. We propose models of increasing complexity and accuracy, ranging from queueing networks to stochastic well formed nets, able to estimate job performance under a number of scenarios of interest, including also unreliable resources. The accuracy of our models is evaluated by considering the TPC-DS industry benchmark running experiments on Amazon EC2 and the CINECA Italian supercomputing center. The results have shown that the average accuracy we can achieve is in the range 9-14%.
Year
DOI
Venue
2016
10.1007/978-3-319-49583-5_47
ALGORITHMS AND ARCHITECTURES FOR PARALLEL PROCESSING, ICA3PP 2016
Keywords
Field
DocType
MapReduce,Performance models
Yarn,Core business,Business analytics,Supercomputer,Computer science,Service-level agreement,Queueing theory,Job performance,Performance prediction,Distributed computing
Conference
Volume
ISSN
Citations 
10048
0302-9743
9
PageRank 
References 
Authors
0.56
23
6
Name
Order
Citations
PageRank
danilo ardagna11944112.08
Simona Bernardi231523.24
Eugenio Gianniti3163.77
Soroush Karimian Aliabadi4110.94
Diego Perez-Palacin512013.23
José Ignacio Requeno6416.92