Paper Info
Title
Software performance testing using covering arrays: efficient screening designs with categorical factors
Abstract
Classical Design of Experiment (DOE) techniques have been in use for many years to aid in the performance testing of systems. In particular fractional factorial designs have been used in cases with many numerical factors to reduce the number of experimental runs necessary. For experiments involving categorical factors, this is not the case; experimenters regularly resort to exhaustive (full factorial) experiments. Recently, D-optimal designs have been used to reduce numbers of tests for experiments involving categorical factors because of their flexibility, but not necessarily because they can closely approximate full factorial results. In commonly used statistical packages, the only generic alternative for reduced experiments involving categorical factors is afforded by optimal designs. The extent to which D-optimal designs succeed in estimating exhaustive results has not been evaluated, and it is natural to determine this. An alternative design based on covering arrays may offer a better approximation of full factorial data. Covering arrays are used in software testing for accurate coverage of interactions, while D-optimal and factorial designs measure the amount of interaction. Initial work involved exhaustive generation of designs in order to compare covering arrays and D-optimal designs in approximating full factorial designs. In that setting, covering arrays provided better approximations of full factorial analysis, while ensuring coverage of all small interactions. Here we examine commercially viable covering array and D-optimal design generators to compare designs. Commercial covering array generators, while not as good as exhaustively generated designs, remain competitive with D-optimal design generators.
Year
DOI
Venue
2005
10.1145/1071021.1071034
WOSP
Keywords
Field
DocType
d-optimal design generator,factorial design,approximate full factorial result,full factorial analysis,full factorial,categorical factor,full factorial design,d-optimal design,software performance testing,full factorial data,better approximation,efficient screening design,set cover,optimal design,fractional factorial design,performance testing,design of experiment,software performance,d optimal design,software testing
Fractional factorial design,Single-subject research,Engineering drawing,Systems engineering,Yates analysis,Categorical variable,Computer science,Factorial,Optimal design,Factorial experiment,Reliability engineering,Design of experiments
Conference
ISBN
Citations 
PageRank 
1-59593-087-6
12
0.60
References 
Authors
6
3
Name
Order
Citations
PageRank
Dean S. Hoskins1211.34
Charles J. Colbourn22726290.04
Douglas C. Montgomery310624.05