New software releases are not defect free, and such defects are expensive to fix once they have been deployed in the field. If a company can predict which components are likely to have more defects, then they can focus their quality assurance (QA) efforts more efficiently, and prepare for a higher incidence of support calls. This paper presents a postmortem case study of the Windows Server 2003 defect rates. The focus is on a predictive model that identifies defect-prone components based on their most recent software development history.
The authors looked at a number of software engineering metrics, in an attempt to correlate the defect rate of each binary with the changes made to that binary during the development process. They found a surprisingly high level of correlation between a set of eight relative change measures and the defect rate of the resulting program. The tone of the paper is that of professional statistics. Each argument is stated clearly, along with associated caveats and exceptions. I felt that this approach showed careful analysis, and improved the overall credibility of this paper.
This paper’s contribution is its argument that “code that changes many times pre-release will likely have more post-release defects than code that changes less over the same period of time.” This is a very intuitive notion, and the data and statistics presented in the paper make a strong case.
One of my problems with this paper is that it is focused on proving its main thesis, and not on providing direction to developers or organizations. The metrics are shown to have predictive power, but I was left wondering how to apply them. Does modifying a file 100 times increase the error rate by one percent or 50 percent? How much change is acceptable during a development cycle, and what are the implications?
If you are a researcher in this field, then this paper is probably going to be on your citation list. It is a good, careful case study. If you are a practitioner, then there is little to gain from reading this paper.
