Paper Info
Title
Ibd1: The Metrics And Evaluation Method For Dnn Processor Benchmark While Doing Inference Task
Abstract
With the many varieties of AI hardware prevailing on the market, it is often hard to decide which one is the most suitable to use but not only with the best performance. As there is an industry-wide trend demand for deep learning deployment, the inference benchmark for the effectiveness of DNN processor becomes important and is of great help to select and optimize AI hardware. To systematically benchmark deep learning deployment platforms, and give more objective and useful metrics comparison. In this paper, an end to end benchmark evaluation system was brought up called IBD, it combined 4 steps include three components with 6 metrics. The performance comparison results are obtained from the chipsets from Qualcomm, HiSilicon, and NVIDIA, which can provide hardware acceleration for AI inference. To comprehensively reflect the current status of theDNNprocessor deploying performance, we chose six devices from three kinds of deployment scenarios which are cloud, desktop and mobile, ten models from three different kinds of applications with diverse characteristics are selected, and all these models are trained from three major training frameworks. Several important observations were made by using our methodologies. Experimental results showed that workload diversity should focus on the difference came from training frameworks, inference frameworks with specific processors, input size and precision (floating and quantized).
Year
DOI
Venue
2021
10.3233/JIFS-202552
JOURNAL OF INTELLIGENT & FUZZY SYSTEMS
Keywords
DocType
Volume
AI, deep neural network processor, benchmark, end to end, inference
Journal
40
Issue
ISSN
Citations 
5
1064-1246
0
PageRank 
References 
Authors
0.34
0
4
Name
Order
Citations
PageRank
Weimin Zhang17221.91
Long Zhang200.34
Zheyu Zhang300.34
Mingjun Sun400.34