Computing Reviews

On the techniques we create, the tools we build, and their misalignments: a study of KLEE
Rizzi E., Elbaum S., Dwyer M.  ICSE 2016 (Proceedings of the 38th International Conference on Software Engineering, Austin, TX,  May 14-22, 2016) 132-143, 2016. Type: Proceedings
Date Reviewed: 12/14/16

Software maintenance research seems to gets no respect and it is a minor miracle that the academic publish-or-perish system produced the data in this paper. This data indicates that ignoring maintenance distorts some academic research. The authors took an open-source software tool (the KLEE test generator) and applied half-a-dozen fairly obvious bug fixes and tweaks. The resulting version performed ten times faster! They then looked at the corpus of 100 research papers that mention the tool. None mentioned the researchers’ fixes. This does not surprise me because novelty is rewarded more than repair. Next, they tried to see if the published results would be changed if the maintenance was done first. Seventy-four papers referred to KLEE without modifying it. Twelve papers were robust enough to not need replication. In two papers, it was possible to replicate the research using the properly maintained code. Further, in six papers they could approximately duplicate it. This showed that in seven out of these eight cases, the conclusions would have been changed. In other words, doing maintenance before doing a novel change would have been a good idea.

The paper makes some recommendations: reward the publication and review of artifacts as well as papers, add special tracks in conferences for maintenance, provide institutional support for maintaining software, and so on.

The authors only studied KLEE. I suspect similar results hold elsewhere. It would be good if other groups could replicate the authors’ methodology on other published code.

Reviewer:  Richard Botting Review #: CR144968 (1702-0137)

Reproduction in whole or in part without permission is prohibited.   Copyright 2017™
Terms of Use
| Privacy Policy