Software testing is an exercise in extrapolation--we attempt to infer the correctness of software by exercising it using test cases. However, as Dijkstra famously observed, testing demonstrates the presence of bugs, not their absence. Thus, in some measure, extrapolating from testing results the confidence we have in the correctness of the software begs the question.

This paper provides a very interesting approach for providing numerical measures of confidence in software by extrapolating the results of a testing campaign. The paper takes inspiration from the field of ecology, where statistics is used to provide estimates of the diversity of species. The author makes an analogy between discovering a new species and discovering bugs in a program as part of a fuzzing campaign. The paper provides techniques to estimate various measures of confidence: the defect discovery probability, coverage, and the more interesting effort estimates to achieve a given measure of defect discovery probability. These estimates provide techniques for extrapolating measures of confidence in the correctness of a program from test results. They are a scientific guide to rational decision making about the testing process.

The ideas presented in the paper are illustrated on a group of open-source programs using the AFL fuzzing tool. The author tries to explain any threats to the validity of the presented approach; this does provide some pointers to the applicability of the techniques to other testing approaches, for example, manual testing. However, adapting the techniques to other testing approaches may require a deeper knowledge of statistical methods.