Name
Abstract | ||
|---|---|---|
As a fundamental task in computer architecture research, performance comparison has been continuously hampered by the variability of computer performance. In traditional performance comparisons, the impact of performance variability is usually ignored (i.e., the means of performance observations are compared regardless of the variability), or in the few cases directly addressed with -statistics without checking the number and normality of performance observations. In this paper, we formulate a performance comparison as a statistical task, and empirically illustrate why and how common practices can lead to incorrect comparisons. We propose a non-parametric hierarchical performance testing (HPT) framework for performance comparison, which is significantly more practical than standard -statistics because it does not require to collect a large number of performance observations in order to achieve a normal distribution of sample mean. In particular, the proposed HPT can facilitate quantitative performance comparison, in which the performance speedup of one computer over another is statistically evaluated. Compared with the HPT, a common practice which uses geometric mean performance scores to estimate the performance speedup has errors of to percent on SPEC CPU2006 or SPEC MPI2007, which demonstrates the necessity of using appropriate sta- istical techniques. This HPT framework has been implemented as an open-source software, and integrated in the PARSEC 3.0 benchmark suite. |
Year | DOI | Venue |
|---|---|---|
2015 | 10.1109/TC.2014.2315614 | Computers, IEEE Transactions |
Keywords | Field | DocType |
$t$ -statistics,performance comparison,hierarchical performance testing,performance distribution,computer performance,normal distribution,computer architecture,public domain software,reliability,statistical analysis,benchmark testing,parallel processing,probability distribution,t statistics | Normal distribution,Computer performance,Computer science,Parallel computing,Real-time computing,Software performance testing,Probability distribution,Software,Spec#,Benchmark (computing),Speedup | Journal |
Volume | Issue | ISSN |
64 | 5 | 0018-9340 |
ISBN | Citations | PageRank |
978-1-4673-0825-0 | 18 | 0.92 |
References | Authors | |
14 | 7 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Chen Tianshi | 1 | 1205 | 59.29 |
| Qi Guo | 2 | 169 | 17.04 |
| Olivier Temam | 3 | 2474 | 148.79 |
| Y. Wu | 4 | 1178 | 139.36 |
| Yungang Bao | 5 | 361 | 31.11 |
| Zhiwei Xu | 6 | 38 | 5.61 |
| Yunji Chen | 7 | 1432 | 79.99 |