Title
AutoExecutor: predictive parallelism for spark SQL queries
Abstract
AbstractRight-sizing resources for query execution is important for cost-efficient performance, but estimating how performance is affected by resource allocations, upfront, before query execution is difficult. We demonstrate AutoExecutor, a predictive system that uses machine learning models to predict query run times as a function of the number of allocated executors, that limits the maximum allowed parallelism, for Spark SQL queries running on Azure Synapse.
Year
DOI
Venue
2021
10.14778/3476311.3476362
Hosted Content
DocType
Volume
Issue
Journal
14
12
ISSN
Citations 
PageRank 
2150-8097
1
0.36
References 
Authors
0
7
Name
Order
Citations
PageRank
Rathijit Sen145.50
Abhishek Roy223.08
Alekh Jindal311.37
Rui Fang410.36
Jeff Zheng510.70
Xiaolei Liu6118.70
Ruiping Li743.15