Computing Reviews
Today's Issue Hot Topics Search Browse Recommended My Account Log In
Review Help
Search
An analysis of human factors and label accuracy in crowdsourcing relevance judgments
Kazai G., Kamps J., Milic-Frayling N. Information Retrieval16 (2):138-178,2013.Type:Article
Date Reviewed: Aug 20 2013

Labeling is a complex task. This interesting paper investigates the use of crowdsourcing to create labeled data, which involves relevance judgments that are known to be subjective. The authors use a series of experiments using Amazon’s Mechanical Turk (AMT) to explore the human characteristics of the crowd involved in a relevance assessment task. The study considers three factors or variables--the payment offered to the assessor, the effort expended by the assessor, and the assessor’s qualifications--and examines the effect of these variables on the resulting relevance labels.

The authors also sought to understand how various human factors affect or relate to label accuracy. This interest produced five questions, including: are workers who complete human intelligence tasks (HITs) for fun more accurate in assigning relevance labels than others, and are workers who are happy with their pay more reliable in their work?

The study also explored the characteristics of crowd workers in different task conditions. The authors wanted to find out whether different pay levels attract people with different motivations, and whether better-paid workers find the tasks more interesting. They statistically analyzed answers from the respondents to test for average label accuracy. The study produced several stimulating findings regarding characteristics of the crowd participants. First, for lower-pay HITs, more workers reported fun as the motivating factor. Second, there were no significant differences between levels of self-reported expertise or interest in the task across the lower and higher pay conditions. Third, higher pay led to higher satisfaction with the pay.

The paper may interest those practitioners who are looking for a framework to guide them in the design of crowdsourcing tasks to maximize label quality.

Reviewer:  Amit Rudra Review #: CR141481 (1311-1025)
Bookmark and Share
 
Human Factors (H.1.2 ... )
 
 
Human Information Processing (H.1.2 ... )
 
 
Relevance Feedback (H.3.3 ... )
 
Would you recommend this review?
yes
no
Other reviews under "Human Factors": Date
A theory of computer semiotics
Andersen P., Cambridge University Press, New York, NY, 1990. Type: Book (9780521393362)
Aug 1 1992
An experimental comparison of tabular and graphic data presentation
Powers M., Lashley C., Sanchez P., Shneiderman B. International Journal of Man-Machine Studies 20(6): 545-566, 1984. Type: Article
May 1 1985
Organizing for human factors
Thomas J. (ed)  Human factors and interactive computer systems (, New York,461984. Type: Proceedings
May 1 1985
more...

E-Mail This Printer-Friendly
Send Your Comments
Contact Us
Reproduction in whole or in part without permission is prohibited.   Copyright 1999-2024 ThinkLoud®
Terms of Use
| Privacy Policy