Paper Info
Title
One Size Does Not Fit All: Quantifying and Exposing the Accuracy-Latency Trade-Off in Machine Learning Cloud Service APIs via Tolerance Tiers
Abstract
Today's cloud service architectures follow a “one size fits all” deployment strategy where the same service version instantiation is provided to the end users. However, consumers are broad and different applications have different accuracy and responsiveness requirements, which as we demonstrate renders the “one size fits all” approach inefficient in practice. We use a production grade speech recognition engine, which serves several thousands of users, and an open source computer vision based system, to explain our point. To overcome the limitations of the “one size fits all” approach, we recommend Tolerance Tiers where each MLaaS tier exposes an accuracy/responsiveness characteristic, and consumers can programmatically select a tier. We evaluate our proposal on the CPU-based automatic speech recognition (ASR) engine and cutting-edge neural networks for image classification deployed on both CPUs and GPUs. The results show that our proposed approach provides a MLaaS cloud service architecture that can be tuned by the end API user or consumer to outperform the conventional “one size fits all” approach.
Year
DOI
Venue
2019
10.1109/ISPASS.2019.00012
2019 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)
Field
DocType
ISBN
Architecture,Central processing unit,Software deployment,End user,Latency (engineering),Computer science,Artificial intelligence,Contextual image classification,Artificial neural network,Machine learning,Cloud computing
Conference
978-1-7281-0746-2
Citations 
PageRank 
References 
1
0.37
30
Authors
5
Name
Order
Citations
PageRank
Matthew Halpern11026.47
Behzad Boroujerdian210.37
Todd Mummert310.37
Evelyn Duesterwald4112188.40
Vijay Janapa Reddi52931140.26