Scientists often write programs for which no traditional oracle exists. Over the years, techniques have emerged that can be used to test such programs. This paper provides detailed explanations of three of these techniques: metamorphic testing exploits knowledge of how changes to inputs affect outputs; assertion checking involves embedding constraints into source code; and machine learning involves building classifiers using black-box features to predict whether an output is correct or not. A critical review is also presented of empirical studies that have applied these three techniques.

Several suggestions for future work are made. For example, metamorphic testing was found incapable of killing several mutants in the empirical studies reviewed, and research is needed to determine if it is possible to kill these mutants using stronger metamorphic relations (MRs). While assertion checking was found to be more effective than the null oracle in the empirical study reviewed, it was found to be either very effective or very ineffective depending on the method. Additional empirical studies are needed of the assertion checking technique. The authors conclude that automated support is needed to detect MRs and assertions. Such automated support could significantly reduce the need for domain knowledge required by the test engineer. Addressing falsely signaled faults is wasteful, so any such automated support must also minimize the generation of spurious MRs/assertions.

This paper is a thorough exposition on three techniques that can be used to test the so-called non-testable programs. As such, it is strongly recommended to the software testing community.