crowd guildscrowdsourcingpeer assessmentreputation systemsworker quality
Figures from the paper
Abstract (AI)
Crowd workers are distributed and decentralized. While decentralization is designed to utilize independent judgment to promote high-quality results, it paradoxically undercuts behaviors and institutions that are critical to high-quality work. Reputation is one central example: crowdsourcing systems depend on reputation scores from decentralized workers and requesters, but these scores are notoriously inflated and uninformative. In this paper, we draw inspiration from historical worker guilds (e.g., in the silk trade) to design and implement crowd guilds: centralized groups of crowd workers who collectively certify each other's quality through double-blind peer assessment. A two-week field experiment compared crowd guilds to a traditional decentralized crowd work model. Crowd guilds produced reputation signals more strongly correlated with ground-truth worker quality than signals available on current crowd working platforms, and more accurate than in the traditional model.
Key Findings
1
A two-week field experiment compared crowd guilds with a traditional decentralized crowd work model.
2
Crowd guilds generated reputation signals more strongly correlated with ground-truth worker quality than signals on existing crowd work platforms.
3
Reputation signals from crowd guilds were more accurate than those produced by the traditional decentralized model.
4
The approach addresses the problem that decentralized crowdsourcing reputation scores are often inflated and uninformative.
5
The paper introduces crowd guilds, centralized groups of crowd workers that certify one another’s quality through double-blind peer assessment.
Research Object
crowd guilds as centralized groups of crowd workers
Research Subject
the accuracy and informativeness of peer-certified reputation signals for worker quality compared with decentralized crowd work
Publication Details
Publication Date
2017-02-14
Journal
Publisher
ISSN
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest