Paper Info
Title
HadoopWatch: A first step towards comprehensive traffic forecasting in cloud computing
Abstract
This paper presents our effort towards comprehensive traffic forecasting for big data applications using external, light-weighted file system monitoring. Our idea is motivated by the key observations that rich traffic demand information already exists in the log and meta-data files of many big data applications, and that such information can be readily extracted through run-time file system monitoring. As the first step, we use Hadoop as a concrete example to explore our methodology and develop a system called HadoopWatch to predict traffic demand of Hadoop applications. We further implement HadoopWatch in our real small-scale testbed with 10 physical servers and 30 virtual machines. Our experiments over a series of MapReduce applications demonstrate that HadoopWatch can forecast the traffic demand with almost 100% accuracy and time advance. Furthermore, it makes no modification of the Hadoop framework, and introduces little overhead to the application performance.
Year
DOI
Venue
2014
10.1109/INFOCOM.2014.6847920
INFOCOM
Keywords
Field
DocType
public domain software,parallel programming,external-light-weighted file system monitoring,virtual machines,mapreduce applications,physical servers,real-small-scale testbed,log files,hadoopwatch,comprehensive traffic demand forecasting,information extraction,traffic demand prediction,meta-data files,big data applications,telecommunication traffic,big data,meta data,cloud computing
Computer science,Computer network,Cloud computing,Distributed computing
Conference
ISSN
Citations 
PageRank 
0743-166X
18
0.78
References 
Authors
26
6
Name
Order
Citations
PageRank
Yang Peng1211.24
Kai Chen274459.02
Guohui Wang3108860.78
Wei Bai 0001419013.46
Zhiqiang Ma5242.28
Lin Gu61359.73