Computing Reviews
Today's Issue Hot Topics Search Browse Recommended My Account Log In
Review Help
Search
Coverage is not strongly correlated with test suite effectiveness
Inozemtseva L., Holmes R.  ICSE 2014 (Proceedings of the 36th International Conference on Software Engineering, Hyderabad, India, May 31-Jun 7, 2014)435-445.2014.Type:Proceedings
Date Reviewed: Jul 14 2015

Should developers aim to write test suites with high coverage? To answer this question, the relationships between coverage, test suite size, and test suite effectiveness were systematically explored making use of five reasonably large Java programs that had existing master test suites. The tool CodeCover was used to measure statement, decision, and modified condition coverage. The tool PIT was used to generate mutants and to report mutation kills as the effectiveness measure. Test suite size was varied by randomly sampling from the existing master test suites.

A moderate to high correlation was found between effectiveness and the number of test methods in a test suite. A moderate to high correlation was found between effectiveness and coverage when test suite size was ignored. When test suite size was controlled for, the correlation between effectiveness and coverage was found to range from low to moderate. The authors suggest that coverage should not be used as a quality target. Evidence was also found suggesting that the use of complex coverage measures such as modified condition coverage is not justified.

Since manually determining if thousands of mutants are equivalents is a very costly exercise, the authors simply assumed that all mutants not detected by the existing master test suites were equivalent. A sampling strategy should have been adopted to at least gauge the degree to which this assumption was correct.

Overall, there is much in this study to commend over previous research, and this paper is very strongly recommended to the software engineering community.

Reviewer:  Andy Brooks Review #: CR143607 (1509-0787)
Bookmark and Share
  Reviewer Selected
Featured Reviewer
 
 
Testing And Debugging (D.2.5 )
 
 
Metrics (D.2.8 )
 
Would you recommend this review?
yes
no
Other reviews under "Testing And Debugging": Date
Software defect removal
Dunn R., McGraw-Hill, Inc., New York, NY, 1984. Type: Book (9789780070183131)
Mar 1 1985
On the optimum checkpoint selection problem
Toueg S., Babaoglu O. SIAM Journal on Computing 13(3): 630-649, 1984. Type: Article
Mar 1 1985
Software testing management
Royer T., Prentice-Hall, Inc., Upper Saddle River, NJ, 1993. Type: Book (9780135329870)
Mar 1 1994
more...

E-Mail This Printer-Friendly
Send Your Comments
Contact Us
Reproduction in whole or in part without permission is prohibited.   Copyright 1999-2024 ThinkLoud®
Terms of Use
| Privacy Policy