The results of an experiment comparing test-driven development (TDD) with test-last development (TLD) are reported. The subjects were professional Java developers who were asked to implement seven small user stories adapted from Robert Martin’s bowling game problem. The only statistically significant result obtained was that a static code analysis using the PMD tool came out in favor of TDD. Otherwise, no difference was found between TDD and TLD regarding code quality and developer productivity.

The participant dropout rate was very high and usable data was obtained from only 13 out of 31 developers. This meant the statistical analysis was undertaken using groups of seven (TDD) and six (TLD). There is no real surprise that the experiment was found to lack statistical power and that the findings were inconclusive. Also, multiple statistical tests were performed, but no Bonferroni correction was applied. Had such a correction been applied, the result for the static code analysis would no longer be significant.

The box plots concerning productivity reveal differences between developers by a factor of three or more. Much might have been learned had the investigators switched from an inferential analysis to an inductive analysis. For example, could the differences in productivity be attributed to varying levels of developer experience and expertise? A major threat to the validity of the experiment, recognized by the investigators themselves, is the fact that the developers performed the task at their places of work at times that suited them.

This paper is recommended only to those with an interest in TDD.