Computing Reviews
Today's Issue Hot Topics Search Browse Recommended My Account Log In
Review Help
On the techniques we create, the tools we build, and their misalignments: a study of KLEE
Rizzi E., Elbaum S., Dwyer M.  ICSE 2016 (Proceedings of the 38th International Conference on Software Engineering, Austin, TX, May 14-22, 2016)132-143.2016.Type:Proceedings
Date Reviewed: Dec 14 2016

Software maintenance research seems to gets no respect and it is a minor miracle that the academic publish-or-perish system produced the data in this paper. This data indicates that ignoring maintenance distorts some academic research. The authors took an open-source software tool (the KLEE test generator) and applied half-a-dozen fairly obvious bug fixes and tweaks. The resulting version performed ten times faster! They then looked at the corpus of 100 research papers that mention the tool. None mentioned the researchers’ fixes. This does not surprise me because novelty is rewarded more than repair. Next, they tried to see if the published results would be changed if the maintenance was done first. Seventy-four papers referred to KLEE without modifying it. Twelve papers were robust enough to not need replication. In two papers, it was possible to replicate the research using the properly maintained code. Further, in six papers they could approximately duplicate it. This showed that in seven out of these eight cases, the conclusions would have been changed. In other words, doing maintenance before doing a novel change would have been a good idea.

The paper makes some recommendations: reward the publication and review of artifacts as well as papers, add special tracks in conferences for maintenance, provide institutional support for maintaining software, and so on.

The authors only studied KLEE. I suspect similar results hold elsewhere. It would be good if other groups could replicate the authors’ methodology on other published code.

Reviewer:  Richard Botting Review #: CR144968 (1702-0137)
Bookmark and Share
  Featured Reviewer  
Tools (D.2.1 ... )
Software Libraries (D.2.2 ... )
Software/ Program Verification (D.2.4 )
Would you recommend this review?
Other reviews under "Tools": Date
C-TODOS: an automatic tool for office system conceptual design
Pernici B., Barbic F., Maiocchi R., Fugini M., Rames J., Rolland C. ACM Transactions on Information Systems 7(4): 378-419, 1989. Type: Article
Jul 1 1990
Larch: languages and tools for formal specification
Guttag J. (ed), Horning J., Springer-Verlag New York, Inc., New York, NY, 1993. Type: Book (9780387940069)
May 1 1994
A survey of structured and object-oriented software specification methods and techniques
Wieringa R. ACM Computing Surveys 30(4): 459-527, 1998. Type: Article
Jun 1 1999

E-Mail This Printer-Friendly
Send Your Comments
Contact Us
Reproduction in whole or in part without permission is prohibited.   Copyright 1999-2024 ThinkLoud®
Terms of Use
| Privacy Policy