Paper Info
Title
Improving experimental methods on success rates in evolutionary computation.
Abstract
Due to the complexity of theoretical approaches in evolutionary computation (EC), research has being largely performed on experimental basis. One popular measure used by the EC community is the success rate (SR), which is used alone or as part of more complex measures such as Koza's computational effort in genetic programming. A common practice in EC is to report just a punctual estimation of the SR, without additional information about its associated uncertainty. We aim to motivate EC researchers to adopt more rigorous practices when working with SRs. In particular, we introduce the importance of correctly reporting this measure and highlight its binomial nature. Unfortunately, this fact is usually overlooked in the literature. Considering the binomiality of the SR opens the whole corpus of binomial statistics to EA research and practice. In particular, we focus on studying several methods to compute SR confidence intervals, the factors that determine their quality in terms of coverage probability and interval length. Due to its practical interest, we also briefly discuss the number of required runs to build confidence intervals with a certain quality, providing a sound method to set the number of runs, one of the most important experimental settings in EC. Evidence suggests that Wilson is, on average, a reliable and simple method to bound an estimation of SR with confidence intervals, while the standard method, which is quite popular because of its conceptual simplicity, should be avoided in any case. However, other methods can also be of interest under certain circumstances. We encourage to report the number of trials and successes, as well as the interval, to ease further comparability of the results.
Year
DOI
Venue
2017
10.1080/0952813X.2016.1214186
JOURNAL OF EXPERIMENTAL & THEORETICAL ARTIFICIAL INTELLIGENCE
Keywords
Field
DocType
Experimentation,success rate,confidence intervals,genetic programming,evolutionary computation,performance measures
Computer science,Binomial,Evolutionary computation,Genetic programming,Artificial intelligence,Confidence interval,Machine learning
Journal
Volume
Issue
ISSN
29.0
4
0952-813X
Citations 
PageRank 
References 
0
0.34
18
Authors
3
Name
Order
Citations
PageRank
David F. Barrero112017.17
María D. R-Moreno29715.22
David Camacho333143.45