Paper Info
Title
Attention Please: Consider Mockito when Evaluating Newly Released Automated Program Repair Techniques.
Abstract
Automated program repair (APR) has attracted widespread attention in recent years with substantial techniques being proposed. Meanwhile, a number of benchmarks have been established for evaluating the performances of APR techniques, among which Defects4J is one of the most widely used benchmark. However, bugs in Mockito, a project augmented in a later-version of Defects4J, do not receive much attention by recent researches. In this paper, we aim at investigating the necessity of considering Mockito bugs when evaluating APR techniques. Our findings show that: 1) Mockito bugs are not more complex for repairing compared with bugs from non-Mockito projects; 2) the bugs repaired by the state-of-the-art tools share the same repair patterns compared with those patterns required to repair Mockito bugs; however, 3) the state-of-the-art tools perform poorly on Mockito bugs (Nopol can only correctly fix one bug while SimFix and CapGen cannot fix any bug in Mockito even if all the buggy locations have been exposed). We conclude from these results that existing APR techniques may be overfitting to their evaluated subjects and we should consider Mockito, or even more bugs from other projects, when evaluating newly proposed APR techniques.
Year
Venue
Field
2018
arXiv: Software Engineering
Mockito,Software engineering,Systems engineering,Computer science,Overfitting
DocType
Volume
Citations 
Journal
abs/1812.05370
2
PageRank 
References 
Authors
0.35
18
4
Name
Order
Citations
PageRank
Shangwen Wang120.69
Ming Wen221.03
Deheng Yang320.69
Xiaoguang Mao432.73