Paper Info
Title
Automatic detection of performance deviations in the load testing of large scale systems
Abstract
Load testing is one of the means for evaluating the performance of Large Scale Systems (LSS). At the end of a load test, performance analysts must analyze thousands of performance counters from hundreds of machines under test. These performance counters are measures of run-time system properties such as CPU utilization, Disk I/O, memory consumption, and network traffic. Analysts observe counters to find out if the system is meeting its Service Level Agreements (SLAs). In this paper, we present and evaluate one supervised and three unsupervised approaches to help performance analysts to 1) more effectively compare load tests in order to detect performance deviations which may lead to SLA violations, and 2) to provide them with a smaller and manageable set of important performance counters to assist in root-cause analysis of the detected deviations. Our case study is based on load test data obtained from both a large scale industrial system and an open source benchmark application. The case study shows, that our wrapper-based supervised approach, which uses a search-based technique to find the best subset of performance counters and a logistic regression model for deviation prediction, can provide up to 89% reduction in the set of performance counters while detecting performance deviations with few false positives (i.e., 95% average precision). The study also shows that the supervised approach is more stable and effective than the unsupervised approaches but it has more overhead due to its semi-automated training phase.
Year
DOI
Venue
2013
10.1109/ICSE.2013.6606651
Software Engineering
Keywords
Field
DocType
large scale system,performance analyst,load test data,load testing,unsupervised approach,case study shows,case study,automatic detection,load test,performance deviation,performance counter,important performance counter,public domain software,machine learning,cpu utilization,radiation detectors,logistics,regression analysis,unsupervised learning,principal component analysis,root cause analysis,logistic regression model,performance,testing,control charts,signature
Data mining,Service level,Load testing,Regression analysis,CPU time,Computer science,Industrial systems,Real-time computing,Software performance testing,Unsupervised learning,False positive paradox
Conference
Volume
ISBN
Citations 
2
978-1-4673-3076-3
37
PageRank 
References 
Authors
1.06
11
3
Name
Order
Citations
PageRank
Haroon Malik120216.17
Hadi Hemmati262227.54
Ahmed E. Hassan35959287.68