Name
Abstract | ||
|---|---|---|
Automated tests play an important role in software evolution because they can rapidly detect faults introduced during changes. In practice, code-coverage metrics are often used as criteria to evaluate the effectiveness of test suites with focus on regression faults. However, code coverage only expresses which portion of a system has been executed by tests, but not how effective the tests actually are in detecting regression faults.
Our goal was to evaluate the validity of code coverage as a measure for test effectiveness. To do so, we conducted an empirical study in which we applied an extreme mutation testing approach to analyze the tests of open-source projects written in Java. We assessed the ratio of pseudo-tested methods (those tested in a way such that faults would not be detected) to all covered methods and judged their impact on the software project. The results show that the ratio of pseudo-tested methods is acceptable for unit tests but not for system tests (that execute large portions of the whole system). Therefore, we conclude that the coverage metric is only a valid effectiveness indicator for unit tests.
|
Year | DOI | Venue |
|---|---|---|
2016 | 10.1145/2896941.2896944 | CSED@ICSE |
Keywords | DocType | Volume |
Regression Testing, Test Suite Effectiveness, Code Coverage, Mutation Testing | Conference | abs/1611.07163 |
ISSN | ISBN | Citations |
Proceedings of the International Workshop on Continuous Software
Evolution and Delivery (CSED '16). ACM, 2016 | 978-1-4503-4157-8 | 3 |
PageRank | References | Authors |
0.39 | 5 | 3 |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Rainer Niedermayr | 1 | 5 | 1.76 |
| Elmar Juergens | 2 | 743 | 31.07 |
| Stefan Wagner | 3 | 109 | 29.08 |