Paper Info
Title
Variability and Reproducibility in Software Engineering: A Study of Four Companies that Developed the Same System
Abstract
The scientific study of a phenomenon requires it to be reproducible. Mature engineering industries are recognized by projects and products that are, to some extent, reproducible. Yet, reproducibility in software engineering (SE) has not been investigated thoroughly, despite the fact that lack of reproducibility has both practical and scientific consequences. We report a longitudinal multiple-case study of variations and reproducibility in software development, from bidding to deployment, on the basis of the same requirement specification. In a call for tender to 81 companies, 35 responded. Four of them developed the system independently. The firm price, planned schedule, and planned development process, had, respectively, “low,” “low,” and “medium” reproducibilities. The contractor's costs, actual lead time, and schedule overrun of the projects had, respectively, “medium,” “high,” and “low” reproducibilities. The quality dimensions of the delivered products, reliability, usability, and maintainability had, respectively, “low,” "high,” and “low” reproducibilities. Moreover, variability for predictable reasons is also included in the notion of reproducibility. We found that the observed outcome of the four development projects matched our expectations, which were formulated partially on the basis of SE folklore. Nevertheless, achieving more reproducibility in SE remains a great challenge for SE research, education, and industry.
Year
DOI
Venue
2009
10.1109/TSE.2008.89
IEEE Trans. Software Eng.
Keywords
Field
DocType
planned schedule,software development,development project,se research,se folklore,schedule overrun,scientific consequence,planned development process,mature engineering industry,longitudinal multiple-case study,software engineering,development process,general,life cycle,job shop scheduling,usability,programming,software measurement,software process,software quality
Software deployment,Software engineering,Computer science,Usability,Software development process,Software quality,Bidding,Software development,Maintainability,Software measurement
Journal
Volume
Issue
ISSN
35
3
0098-5589
Citations 
PageRank 
References 
38
1.38
42
Authors
3
Name
Order
Citations
PageRank
Bente C. D. Anda1461.94
Dag I. K. Sjoberg277135.28
Audris Mockus34031308.78