Paper Info
Title
Cherrypick: Adaptively Unearthing The Best Cloud Configurations For Big Data Analytics
Abstract
Picking the right cloud configuration for recurring big data analytics jobs running in clouds is hard, because there can be tens of possible VM instance types and even more cluster sizes to pick from. Choosing poorly can significantly degrade performance and increase the cost to run a job by 2-3x on average, and as much as 12x in the worst-case. However, it is challenging to automatically identify the best configuration for a broad spectrum of applications and cloud configurations with low search cost. CherryPick is a system that leverages Bayesian Optimization to build performance models for various applications, and the models are just accurate enough to distinguish the best or close-to-the-best configuration from the rest with only a few test runs. Our experiments on five analytic applications in AWS EC2 show that CherryPick has a 45-90% chance to find optimal configurations, otherwise near-optimal, saving up to 75% search cost compared to existing solutions.
Year
Venue
Field
2017
PROCEEDINGS OF NSDI '17: 14TH USENIX SYMPOSIUM ON NETWORKED SYSTEMS DESIGN AND IMPLEMENTATION
Computer science,Bayesian optimization,Search cost,Big data,Distributed computing,Cloud computing
DocType
Citations 
PageRank 
Conference
27
1.08
References 
Authors
13
6
Name
Order
Citations
PageRank
Omid Alipourfard11426.03
Hongqiang Liu249725.77
Jianshu Chen388352.94
Shivaram Venkataraman4108263.77
Minlan Yu51855107.25
Ming Zhang63509181.37