Title
Experimenting with software testbeds for evaluating new technologies
Abstract
The evolution of a new technology depends upon a good theoretical basis for developing the technology, as well as upon its experimental validation. In order to provide for this experimentation, we have investigated the creation of a software testbed and the feasibility of using the same testbed for experimenting with a broad set of technologies. The testbed is a set of programs, data, and supporting documentation that allows researchers to test their new technology on a standard software platform. An important component of this testbed is the Unified Model of Dependability (UMD), which was used to elicit dependability requirements for the testbed software. With a collection of seeded faults and known issues of the target system, we are able to determine if a new technology is adept at uncovering defects or providing other aids proposed by its developers. In this paper, we present the Tactical Separation Assisted Flight Environment (TSAFE) testbed environment for which we modeled and evaluated dependability requirements and defined faults to be seeded for experimentation. We describe two completed experiments that we conducted on the testbed. The first experiment studies a technology that identifies architectural violations and evaluates its ability to detect the violations. The second experiment studies model checking as part of design for verification. We conclude by describing ongoing experimental work studying testing, using the same testbed. Our conclusion is that even though these three experiments are very different in terms of the studied technology, using and re-using the same testbed is beneficial and cost effective.
Year
DOI
Venue
2007
10.1007/s10664-006-9034-0
Empirical Software Engineering
Keywords
Field
DocType
Empirical study,Technology evaluation,Software testbed
Dependability,Model checking,Systems engineering,Computer science,Testbed,Software,Emerging technologies,Documentation,Empirical research
Journal
Volume
Issue
ISSN
12
4
1382-3256
Citations 
PageRank 
References 
6
0.51
10
Authors
17
Name
Order
Citations
PageRank
Mikael Lindvall1120395.86
Ioana Rus235233.74
Paolo Donzelli318315.20
Atif M. Memon42380135.27
Marvin Zelkowitz5319.41
Aysu Betin-Can61649.10
Tevfik Bultan72481157.95
Chris Ackermann81074.60
Bettina Anders9101.22
Sima Asgari109911.29
Victor Basili1171581311.66
Lorin Hochstein1229527.62
Jörg Fellmann1360.85
Forrest Shull142231148.90
Roseanne Tvedt1560.51
Daniel Pech16323.08
Daniel Hirschbach1770.86