Web applications can be tested automatically by looking for changes in Hypertext Markup Language (HTML) output using a previous working version of the application as the oracle. Many properties of HTML code are best ignored, however, when looking for changes indicative of failure. For example, the absence of an explicit closing tag may have no practical consequence. So what makes for the best oracle comparator? The authors sought to answer this question by constructing a suite of 22 automated oracle comparators, and performing two experiments on four applications. The first experiment analyzed the effect of nondeterministic/real-time application behavior, and gauged the false positives generated due to, for example, a change of date. The second experiment involved the seeding of faults to assess precision and recall in failure detection.

The results of the first experiment were as expected, with Figure 4 clearly showing how false positives follow the partial ordering of oracles based on the HTML structure and content being considered. The results of the second experiment found that, across all four applications, the Forms-Select comparator is best, though this is not easy to discern from Figure 5 alone. Different comparators are recommended for use to minimize either false negatives or false positives.

The value of this approach to Web application testing in a practical context is, however, far from obvious. New or changing user requirements typically lead to changes in HTML code, and the false positive rate is likely to be unacceptably high. Nevertheless, this paper is recommended to the Web engineering community.