While extensive testing is undoubtedly useful, it is also tedious. Automating this task as much as is feasible is an active research area; it has been more successfully applied (until now) to statically typed compiled languages. This work, while not the first of its kind, significantly advances what can be realistically done for interpreted languages.

By slightly instrumenting a language’s own interpreter, correct coverage of the full language is almost trivial to obtain. A few quite clever optimizations then make the results fairly practical. The main point is to be able to differentiate between low-level branching of the interpreter and high-level branching of the interpreted program: the goal is good coverage of the program, not the interpreter. All of the necessary concepts (symbolic execution, instrumentation, what interpreter optimizations to turn off, and others) for understanding the presented ideas are clearly explained; related work is cited when appropriate. The evaluation section is thorough yet informative.

The paper itself is beautifully written, using clear, concrete, and precise language. At all times, the reader sees the high-level picture that the authors are trying to paint, as well as the details that are presented in each section. It is thoroughly motivated, with deftly analyzed experimental results. This is a significant paper, yet the authors do not overstate their case, carefully inserting appropriate caveats where necessary. This paper is an absolute must-read for anyone remotely interested in interpreted languages.