CrowdCog: A Cognitive Skill based System for Heterogeneous Task Assignment and Recommendation in Crowdsourcing

Danula Hettiachchi, Niels Van Berkel, Vassilis Kostakos, Jorge Goncalves

Research output: Contribution to journalConference article in JournalResearchpeer-review

24 Citations (Scopus)
190 Downloads (Pure)

Abstract

While crowd workers typically complete a variety of tasks in crowdsourcing platforms, there is no widely accepted method to successfully match workers to different types of tasks. Researchers have considered using worker demographics, behavioural traces, and prior task completion records to optimise task assignment. However, optimum task assignment remains a challenging research problem due to limitations of proposed approaches, which in turn can have a significant impact on the future of crowdsourcing. We present 'CrowdCog', an online dynamic system that performs both task assignment and task recommendations, by relying on fast-paced online cognitive tests to estimate worker performance across a variety of tasks. Our work extends prior work that highlights the effect of workers' cognitive ability on crowdsourcing task performance. Our study, deployed on Amazon Mechanical Turk, involved 574 workers and 983 HITs that span across four typical crowd tasks (Classification, Counting, Transcription, and Sentiment Analysis). Our results show that both our assignment method and recommendation method result in a significant performance increase (5% to 20%) as compared to a generic or random task assignment. Our findings pave the way for the use of quick cognitive tests to provide robust recommendations and assignments to crowd workers.
Original languageEnglish
Article number110
JournalProceedings of the ACM on Human-Computer Interaction
Volume4
Issue numberCSCW2
DOIs
Publication statusPublished - 14 Oct 2020
Event23rd ACM Conference on Computer-Supported Cooperative Work and Social Computing: CSCW 2020 - Minneapolis, United States
Duration: 17 Oct 202021 Oct 2020

Conference

Conference23rd ACM Conference on Computer-Supported Cooperative Work and Social Computing
Country/TerritoryUnited States
CityMinneapolis
Period17/10/202021/10/2020

Keywords

  • cognitive abilities
  • crowdsourcing
  • dynamic task assignment

Fingerprint

Dive into the research topics of 'CrowdCog: A Cognitive Skill based System for Heterogeneous Task Assignment and Recommendation in Crowdsourcing'. Together they form a unique fingerprint.

Cite this