When is software testing “done”? The obvious answer is when all the defects have been removed. Knowing how many defects exist in a software product has traditionally hinged on a variety of metrics, such as the use of actual inspection data, or data collected from previous projects, to estimate the defects in a new project. These methods may result in inaccurate estimates, as the historical data are not always available, or may ignore critical factors such as the programming language, developer experience, and complexity.
In this paper, a new method for the estimation of software defect density is described that, first, uses the defect count in the current software product (rather than historical data); second, does not require initial values; and, third, is able to produce estimates based on the defect data collected during testing. This new method, called “estimation of defects based on defect decay model” (ED3M), is fully automated. Although the details of an automated tool to calculate the defect density are not provided, the mathematical basis for the calculations entailed is.
The fundamental and simplifying assumption of the model is that Gaussian noise affects the whole testing process identically and that samples taken at different instances are independent of each other. Details support this assumption, and the mathematical calculations necessary to support the assumption are provided.
The method “has three main components: the estimator, the λ1 λ2 approximator, and the error correction block.” The estimator calculates “an estimate for the total number of defects in the product,” and requires only the defect observations and the initial approximator values. The approximator accepts defect observation data to generate the rate and scale parameters for the defect curve (that is, the number of remaining defects) over time. The third component is used to further refine the estimate:
The accuracy of the estimator is heavily dependent on the data set provided. For example, with few initial points in a data set, it is very difficult to produce highly accurate estimates. Error correction techniques are used to improve the estimate and, consequently, reduce the convergence time.
The case studies used test data from five working days to validate the model.
The ED3M approach to determining the total defects in a software product, as well as the estimate of time to remove a specific number of defects, has been validated using case studies, simulated data sets, and data sets reported in the literature. One important observation noted in the validation of the model is that ED3M estimated more defects than the actual number of defects found. This was attributed to hidden defects in the software that were not uncovered during testing.
This is a well-written, compelling paper, with detailed descriptions of the underlying assumptions and calculations used to derive the model. Software test managers will need a strong mathematical and statistical background, as well as programming expertise, to implement the model proposed in this paper. Hopefully, the authors will provide a future paper with accompanying spreadsheet and programming implementation instructions.
