Dr.Westphal suggests in his rebuttal on morethan one occasion that my criticisms of the newsletter and the field aredirected to all work in knowledgeacquisition. That is not so. Work in computer induction for knowledgeacquisition is progressing nicely, and I gave several references in myreview of work I termed successful.His comments that I did not survey the field of computer induction arenot to the point: mechanized knowledge acquisition was outside the scopeof the newsletter.
I do stand by my comments that “as a human factors field,knowledge elicitation is still at square one.” The qualification,“as a human factors field,” and the use of the term“knowledge elicitation”--which refers toperson-to-person interviewing--were conveniently overlooked.Dr. Westphal generates a lot ofsmoke with many angry words to a gentle, if entirely critical, review,but he fails to provide even one example of a genuinely engineeredmethodology for knowledge acquisition. Not one case was provided whereinterviewing methods have gone substantially beyond “justtalk”: Where are the algorithms and heuristics forinterviewing experts that would be expected--indeeddemanded--of publications in any other area of artificialintelligence? This was my key point: “None [of the interviewingtechniques] offers us an algorithm for interviewing which is effective(even generally) and definite (even up to a point).” Knowledgeelicitation clearly has a long way to go before it even comes close toautomated knowledge acquisition or the rest of the expert systemsfield.
Dr.Westphal refers to four journals and claimsthat I said they were worthless. What I actually said--in thesurvey part of the review--was that these were the four journals inthe field and only one of them has been around for any time(The International Journal of Man-machineStudies). I called these publications “academicjournals of note,” and while my overall comments about the absenceof rigorous methods for interviewing include contributions to thosejournals, there was no imputation on my part that these journals shouldclose up shop, as Dr. Westphal implies.
Dr.Westphal claims that I“indirectly” suggested that “industries, academicinstitutions, and government offices” havewasted (his emphasis) their time intheir attempts at knowledge acquisition. Well, everyone involved,however slightly, in the design of expert systems has heard of“the knowledge acquisition bottleneck,” and the originalreview made very clear that virtually all the newsletter’s contributorsagree that this bottleneck hasoverwhelmed entire expert systemsprojects. Until more effective means of extracting expertise fromexperts come along, the development of expert systems will continue torely heavily on person-to-person interviewing, but this is a mosttime-consuming task and if it could be done with any de gree ofefficiency, lots less time would bewasted (my emphasis).
Dr.Westphal points out that sales of thenewsletter have been good. I wrote my review well over a year prior toits publication. If it had been published earlier, perhaps sales wouldhave been less brisk. At any rate, it is precisely because the field isso bereft of real techniques that break the knowledge acquisitionbottleneck that researchers and practitioners would look forward to newpublications that might offer some progress. Unfortunately, thenewsletter simply failed in that regard.
When I remarked that little had been offered beyond what hadalready been published, I did not, of course, mean thatDr.Westphal accepted reprints of articles, elsewherepublished, for that newsletter. What I meant was, as is apparent fromthe text of the original review, that the techniques offered (such astranscripts, grids, observations during interviewing, and technologicalaids for interviewing (not for actual learning)) did not go beyond whatin various forms had already been published. The pieces may have beenoriginal in the narrowest sense of the term--that appropriate foruse by journal referees--but they are not original in the sense ofadding something genuinely new to the discussion.
I continue to maintain that computer scientists are not specialistsin human factors, and hence have not yet developed the skills necessaryto find algorithms or even heuristics for conducting such tasks asinterviews, tasks not normally conceived of as scientific and notnormally understood to be within the scope of computer science.
Finally, Dr.Westphal is wrong and unreasonableto suggest that I “universally depreciated” the work ofthose in knowledge elicitation. I reviewed the written material beforeme as I have done for ComputingReviews many times before; intellectual honestycompels me to say that the newsletter made no scholarly contribution andprovided no progress in breaking the knowledge acquisitionbottleneck.