Paper Info
Title
An Adaptive Auto-configuration Tool for Hadoop
Abstract
With the coming concept of 'big data', the ability to handle large datasets has become a critical consideration for the success of industrial organizations such as Google, Amazon, Yahoo! and Facebook. As an important Cloud Computing framework for bulk data processing, Hadoop is widely used in these organizations. However, the performance of MapReduce is seriously limited by its stiff configuration strategy. Even for a single simple job in Hadoop, a large number of tuning parameters have to be set by users. This may easily lead to performance loss due to some misconfigurations. In this paper, we present an adaptive automatic configuration tool (AACT) for Hadoop to achieve performance optimization. To achieve this goal, we propose a mathematical model which will accurately learn the relationship between system performance and configuration parameters, then configure Hadoop system based on this mathematical model. With the help of AACT, Hadoop is able to adapt the hardware and software configurations dynamically and drive the system to an optimal configuration in acceptable time. Experimental results show its efficiency and adaptability, and that it is ten times faster compared with default configuration.
Year
DOI
Venue
2014
10.1109/ICECCS.2014.17
ICECCS
Keywords
Field
DocType
software configuration,amazon,hadoop, auto-configuration, self-learning,bulk data processing,big data,hardware configuration,mapreduce,cloud computing framework,industrial organizations,google,mathematical model,optimal configuration,tuning parameters,hadoop,facebook,auto-configuration,performance optimization,aact,adaptive auto-configuration to,yahoo!,social networking (online),cloud computing,adaptive automatic configuration tool,self-learning,hardware,optimization,system performance,benchmark testing
Adaptability,Data processing,Computer science,Real-time computing,Software,Big data,Auto-configuration,Benchmark (computing),Cloud computing,Distributed computing
Conference
Citations 
PageRank 
References 
1
0.35
0
Authors
7
Name
Order
Citations
PageRank
Changlong Li1266.88
Hang Zhuang2266.54
Kun Lu3213.75
Mingming Sun4324.87
Jinhong Zhou593.40
Dai, Dong68816.49
Xuehai Zhou755177.54