In this experiment, 27 versions of a program, all produced by different programmers, were given one million randomly produced sets of input data and their reactions were compared, with special regard to the use of the different versions in an N-version program. In this report, the deviations of the results produced by the program versions from those produced by a carefully designed, so-called “gold” program are analyzed in detail. For each version, the authors explain the deviations detected in terms of a set of faults of the version.

The most noteworthy result of the analysis is that similarity of the patterns of faulty reactions of several program versions does not imply a similarity of faults of the programs. Not surprisingly, the faults found in the program versions were predominantly due to failures by the programmers to handle rounding arithmetic properly so as to obtain correct results in special cases. It is surprising, however, that the programmers turned out in a postexperiment questionnaire to be unaware that this is an especially difficult area of programming. Another significant result is that the authors found no indication that changing development tools or methods, or any other simple technique, would significantly reduce the incidence of correlated failures in N-version software.

The problem solved by the programs studied in the paper relates to the mapping of conditions of the continuously varying world in which we find ourselves onto discrete conditions, such as whether to launch a missile carrying a nuclear warhead. The present report and the authors’ previous paper [1] show in detail some reasons why the establishment of such a mapping in an intuitively convincing manner is extremely difficult. These are important contributions.