An issue facing the new wave of Web applications is just how good the searches on the World Wide Web can be. The authors of this paper did a systematic study of two metrics, the relevance of the returns and the false positives in a search reply, and found that “recent studies have demonstrated that ... improvements in information retrieval system effectiveness ... do not translate into a benefit for [search engine] users.” This is an interesting result, and will be useful for designers of search engines.

In section 1, the significance of the benefit users gain in the instance recall task may undermine the authors’ principal conclusion. The authors also do not describe the 30 users who were test subjects. Did they measure their intelligence? Would a smart user give different results than a stupid one?

I found the paper to be very interesting and quite well done. The graphs are easy to understand, the writing is crisp, and the discussion section is excellent.