Student learning for concepts like automata and formal languages can be a challenge. Students often fail to see the relevance to more practical aspects, and the feedback they receive on lab exercises or homework assignments typically occurs with significant delays, and may offer little meaningful information. From an instructor’s or grader’s perspective, assessing large numbers of similar exercises quickly becomes tedious.

In an attempt to provide real-time, meaningful feedback, the authors developed an automatic assessment system for deterministic finite automata. It provides instant, personalized feedback, allowing students to incrementally work on a problem until they find a correct solution. In addition to binary (correct/incorrect) and counterexamples reported in related work, the authors construct hints for students with suggestions for strategies to improve their current solution.

The system was tested at several universities with hundreds of students in classes on theoretical computer science. While the evaluation emphasis was more on a comparison of the different types of feedback, in comparison with similar exercises by students from earlier classes not using any such tools, there was significant improvement in the student scores.

Binary feedback was helpful, but less so than the other two types, both according to differences in scores, attempts, and time needed to find a correct solution and to student feedback. Preferences between hints and counterexamples were fairly evenly divided among students, and empirical results also were not conclusive.

Considering that it is fairly easy to incorporate such a tool into a class as an additional resource or replacement for some graded exercises, it is not surprising that it has been adopted by a significant number of instructors at various institutions.