This well-written and important (if not well-referenced) paper contains some unusual observations about the risks of developing embedded software products.

I was impressed that the 90 projects the authors analyzed underwent regular design reviews and that the reviews were structured well enough to uncover common risk areas. Sections 3 and 4 recommend specific areas where education improvement for future software design engineers is long overdue. As explained in Section 2.2, reviewers held the design reviews when a “product was ready to start acceptance testing or to be released.“ This is rather late for my taste.

Maranzano et al. detail the merits of earlier architecture reviews [1]. In their article, Maranzano et al. and Koopman agree:

Complex software projects are notoriously late to market, often exhibit quality problems, and don’t always deliver on promised functionality. In our companies, we have observed too many projects that either fail or require significant rework late in their schedules. Line managers and engineers can’t communicate problems consistently up the chain of command because they fear reprisal or they don’t have the knowledge or expertise to detect a significant issue. [1]

I have never seen that “five forebodes failure,” meaning that software development teams with exactly five primary members have the “most spectacular project failures,” as observed in Section 4.6. We know that small is beautiful, and that smaller teams produce clever software.

Tiny teams seem to operate more like artisans than professional engineers, and Koopman’s anecdotal finding--as projects grow, they need more structure--is well understood. Even though the finding is not news, it is clearly stated, and it marks the transition of a project from a master craftsman approach to an industrial one. We know that we can make anything work once. But industrial-strength shops need to make success repeatable, and not dependent on a few indispensable people.

I found the second risk area of “not enough paper” (Section 3.1) surprising. Abundant writing exists about the burden of documentation, yet Koopman’s observation supports the agile idea that the appropriate level of documentation is an engineering decision. You can take his design findings (Section 3.3) to the bank: they apply to all industrial-strength software efforts.

This paper would be stronger if it referenced more software risk literature. It presents a list of risk factors similar to that published by Boehm [2,3] and Basili [3,4]. The classification of risk areas and principle findings could follow Maranzano et al. or Boehm.

I was shocked to find that two major risks are missing: schedules are often too tight, and, in 2007 and 2008, an embedded systems design team’s greatest concern was staying on schedule [5]; the other missing risk is that incompetent and/or inexperienced managers can jeopardize the work of the project team. Since these results are for tiny teams, the effects of dictated schedules and management shortfalls may not be as significant as for large projects.

If you have a small, elite software shop that you depend on to deliver embedded software, read this paper carefully, heeding the risks identified. Have the mindset that you must prove out the risk area with data, and beware the typical mindset that you must prove in risk mitigation.

Finally, ignore Koopman’s risk list at your own risk.