I approach research papers from a self-centered perspective: how can I use this information today? Immediately applicable ideas with downloadable code rank the highest with me. Surveys of existing research are interesting, but they get tucked away for later and are less appreciated. Research that shows promise but needs more work ranks the lowest. This is an exhaustive and very well-written paper, but it falls into that “needs more work” category and, therefore, off my radar.

The authors are focused on improving the quality of software in commercial embedded devices such as smartphones. They implemented new testing techniques in their workplace that relied primarily on two tactics: analyze data flow across application layers and between tasks, and create test oracles (modules, assertions, and so on) based on analysis of internal system properties (instead of program output, which can be non-deterministic). Data-flow analysis starts with an automated scanner that identifies shared data objects (for example, global variables), but then requires manual follow-up. Enhanced test oracles require manual labor (study of the software) up front to identify quantifiable system properties (such as matched critical section entry and exit, or the state of a binary semaphore), but some of these oracles can be reused across systems and platforms later and become automated.

The value of this paper lies in its analysis of testing techniques. The authors’ approach differs from typical programmer-written testing and black-box feature-based testing, in that they offer an extensive compare-and-contrast postmortem of their empirical findings. This data should energize practicing test engineers. Common test methods that depend on externally observable faults rarely identify deep, internal errors that surface only when the customer uses the product extensively. The test methods proposed by the authors complement traditional methods, dig deeper into unexplored territory, and suggest a path to improving product quality.

The paper is rather long at 24 pages. It is very well written, but the technical depth does require attention to detail; multiple readings will probably be necessary. Tables, charts, and algorithm pseudocode complement the text nicely, and the format is very clean and clear.

This paper will be most useful to a software test engineer interested in expanding both her perspective and toolkit. The average application programmer isn’t invested enough in testing techniques to pursue these ideas. Nothing here is plug-and-play, but the ideas presented are fertile seeds that, with proper consideration, can yield valuable fruit.