In software testing, it is often impractical to calculate expected outputs. Writing assertions is one way of tackling the oracle problem, but do they really detect many more faults than the null oracle, which relies on a program crashing or failing to terminate?

This paper experimentally assesses the effectiveness of Java Modeling Language (JML) assertions in 9 Java classes, by evaluating how well input-side-only tests kill traditional mutations generated with the muJava tool. To simulate testing in industry, tests were written to satisfy branch coverage. On a per-method basis, the median effectiveness of the null oracle was around 11 percent and the median effectiveness of the JML assertions was 61 percent after excluding mutants caught by the null oracle. On a per-method basis, figure 5 shows that JML assertions were either very effective or very ineffective. Data mining was undertaken to understand this bimodality. The decision tree in figure 7 shows that JML assertions were very effective with small methods (≤2 statements).

Data analysis and interpretation correctly proceeded on a per-method basis when an outlier was detected. Readers should be convinced that writing assertions greatly improves fault detection.

The authors advise engineers to write assertions only for noncomplex methods. Since writing assertions that reason fully about the state of a program is likely easier when a method is small and uncomplicated, the quality of the JML assertions should have been scrutinized from this perspective.

Despite this criticism, I strongly recommend the paper to those working in software development and those researching software testing.