The testing tool AutoTest uses contracts in Eiffel code as an oracle, with random test inputs generated automatically. Such testing, though, lacks determinism. How often should void methods be called to force objects into more interesting states (probability PDIV)? How often should a new object be created rather than making use of an existing object (probability PGENNEW)? How often should unrestricted random choices be made for basic types rather than using pre-specified values (probability of generating values from basic types randomly (PGenBasicRand))? To answer these questions, an experiment was undertaken involving the random testing of eight Eiffel classes, using three different seeds for random number generation and all combinations of five possible values (0, 0.25, 0.5, 0.75, 1.0) for PDIV, PGENNEW, and PGenBasicRand. An individual test lasted 30 minutes. Both contract violations and uncaught exceptions were counted as bugs.

The combination that gave the best overall bug finding results (C0) was: PDIV = 0.5, PGENNEW = 0.25, and PGenBasicRand = 0.25. Parsing saved test logs revealed that most bugs were found in the first five minutes. Seed data indicated that it is unwise to rely on the results from a single seed. Do the values for C0 result in better code coverage? No analysis is presented. How strong a guide the combination C0 will prove to be is unclear.

That bugs were found in classes of an Eiffel library clearly points to the fact that random testing, with the use of contracts as an oracle, is effective. This paper is strongly recommended to the software engineering community.