With search engines, users usually examine the top ten search results. In this paper, the authors present a language-model-based approach to re-ranking search results to improve precision at the top five and top ten rank positions. They do this by re-ranking the top 50 search results. The study utilizes inter-passage and inter-document similarities. The authors base the study on the fact that a long or heterogeneous relevant document may contain parts (passages) that are not relevant to the query. Earlier passage-based studies addressed this issue, but they did not consider similarity relationships between documents or between passages. Their model integrates document-query, passage-query, inter-document, and inter-passage similarities.

In the experiments that this paper discusses, the authors use five different Text Retrieval Conference (TREC) test collections. The collections show variety in terms of the number of documents they contain, the nature of the documents (such as news or Web documents), and the length of the documents. It is advantageous to have such variety to see the effectiveness of a model under different conditions. In this detailed study, the authors show that, in several cases, their model statistically and significantly outperforms many other methods.

The TREC information retrieval test collections are an excellent research tool: they provide a lab environment where researchers can repeat experiments and compare the results of different studies. While reading this paper, and while conducting similar activities in my own research, a few questions came to mind: What would happen if we applied this experiment in real life? Would the users feel or appreciate the difference at a statistically significant level (such as a top five precision improvement from 33.9 to 37.1, as shown in Table 1 with the TREC WT10G collection) while using the test collection?

Incidentally, WT10G is the most challenging test collection used in the experiments. The paper provides much better improvements for some of the other collections.

While it would be beneficial to add an actual user dimension to the experiments, and eventually to the test collections, doing so is easier said than done.