Users often receive search results that are not accurate or consistent with their expectations. In this paper, the authors address the challenge of automating document identification, focusing on user needs. Their method improves the search relevance for educational documents by separating documents that have educational value from those that do not. Thus, this research impacts academic search results through a mechanism providing automatic identification of online learning resources.

The considered dataset includes 14 topics from data structures and algorithm courses. A set of features, including relevance, content categories, resource type, expertise, and educational value, is associated with each educational document. Then, an agreement study is carried out to measure how much annotators’ opinions diverge from one another. The main experiment details the use of three automatic classifiers--na¿ve Bayes, support vector machine, and support vector regression--to annotate educational documents. Results prove that the raw content of a document can be used to predict a document’s educational value. One of the key points of the discussion is the evaluation of the contribution of each “user feature” to the classification accuracy. Here, the authors measure the information gain per feature by including manual annotations.