Paper Info
Title
Swift machine learning model serving scheduling: a region based reinforcement learning approach
Abstract
The success of machine learning has prospered Machine-Learning-as-a-Service (MLaaS) - deploying trained machine learning (ML) models in cloud to provide low latency inference services at scale. To meet latency Service-Level-Objective (SLO), judicious parallelization at both request and operation levels is utterly important. However, existing ML systems (e.g., Tensorflow) and cloud ML serving platforms (e.g., SageMaker) are SLO-agnostic and rely on users to manually configure the parallelism. To provide low latency ML serving, this paper proposes a swift machine learning serving scheduling framework with a novel Region-based Reinforcement Learning (RRL) approach. RRL can efficiently identify the optimal parallelism configuration under different workloads by estimating performance of similar configurations with that of the known ones. We both theoretically and experimentally show that the RRL approach can outperform state-of-the-art approaches by finding near optimal solutions over 8 times faster while reducing inference latency up to 79.0% and reducing SLO violation up to 49.9%.
Year
DOI
Venue
2019
10.1145/3295500.3356164
Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
Keywords
Field
DocType
machine-learning-as-a-service (MLaaS), model inference, parallelism parameter tuning, reinforcement learning, service-level-objective (SLO), workload scheduling
Swift,Scheduling (computing),Computer science,Distributed computing,Reinforcement learning
Conference
ISBN
Citations 
PageRank 
978-1-4503-6229-0
3
0.39
References 
Authors
0
6
Name
Order
Citations
PageRank
Heyang Qin131.07
Syed Zawad230.73
Yanqi Zhou351.52
Lei Yang419437.52
Dongfang Zhao536226.49
Feng Yan616318.78