Name
Title | ||
|---|---|---|
Data‐driven benchmarking in software development effort estimation: The few define the bulk |
Abstract | ||
|---|---|---|
AbstractAbstractContextThe rapid evolvement of software development effort estimation models created the need for empirical evaluation of their quality. The empirical evaluation is based either on hypothesis tests with respect to a single criterion or on aggregating methods for multiple criteria. However, a model can be considered as a multidimensional entity performing differently on alternative datasets and its performance can be divergent when expressed by alternative criteria.ObjectiveIn this study, we explore this multidimensional nature of models by considering them as points in two different spaces (domain and criteria spaces).MethodIntroducing an alternative approach for data‐driven benchmarking, a new framework based on archetypal analysis is proposed for evaluation purposes of multiple models.ResultsThe benefits of the framework are illustrated through a large‐scale experimental setup on a set of 93 effort estimation models, trained and tested on 10 datasets under 8 criteria providing answers to critical research questions.ConclusionThe results indicate that a small minority of reference models is enough to define the performance of the bulk of all models. The framework focuses on models that have behavior close to archetypes and especially those that are close to a “best” archetype.In this paper, we propose a data‐driven approach for the evaluation of the multifaceted nature of SDEE models' performance summarized in a three‐step process: (a) exploration of all feasible solutions for the extraction of few reference profiles, (b) characterization of the reference profiles, and (c) evaluation of candidate models in the basis of the extracted reference profiles. The empirical findings indicate that a small minority of reference models is enough to define the performance of the bulk of all models. View Figure |
Year | DOI | Venue |
|---|---|---|
2020 | 10.1002/smr.2258 | Periodicals |
Keywords | DocType | Volume |
archetypal analysis,benchmarks,performance measures,software development effort estimation | Journal | 32 |
Issue | ISSN | Citations |
9 | 2047-7473 | 0 |
PageRank | References | Authors |
0.34 | 0 | 2 |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Nikolaos Mittas | 1 | 238 | 15.03 |
| Lefteris Angelis | 2 | 1296 | 82.51 |