Three approaches to evaluating decision support and expert systems are presented: subjective, technical, and empirical. Subjective evaluation assesses the decision support or expert system from the perspective of the system’s users and sponsors. For subjective evaluation, the author presents several techniques including multiattribute utility technology, cost-benefit analysis, and decision analysis. Technical evaluation determines whether the delivered system is a good technical product. Technical evaluation techniques include software testing methods and a collection of software cost estimation techniques. Finally, empirical evaluation determines whether the users perform better using the new system. The empirical approaches the author presents include experimentation, quasi-experimentation, and case studies.

All three approaches to evaluation are needed because it is easy to see how a system can be delivered that may satisfy one evaluation without satisfying the other two. For example, users may be pleased with a system that is technically inadequate and does not really help them perform their jobs any better. Adelman describes how evaluation needs to be performed throughout the life cycle and shows which of the three evaluation methods is strongest at each stage in the life cycle. The focus of the book, however, is on evaluation of the completed system.

This book has two strengths. First, it is an excellent survey of a wide range of evaluation techniques, from multiattribute utility technology to experiment design. It is well written and extremely readable, and the text is supported by ample references for further study. Second, the book provides a taxonomy of evaluation techniques and shows how those techniques apply to different stages in the development cycle and to different aspects of evaluation. One minor weakness in the book is that it relies heavily on multiattribute utility theory, which may be a burden for less quantitatively oriented evaluators. At the same time, the author makes only a fleeting reference to the analytical hierarchy process[1], which is a much more intuitive model for subjective evaluations.

I recommend this book as a textbook for a course in the evaluation of decision support and expert systems. I also recommend it for practitioners in the field. Although the book focuses on decision support and expert systems, its survey approach and the taxonomies provided would also make it useful to those interested in the evaluation of information systems.