Skip to main content

Award: CrowdCO-OP: Sharing Risks and Rewards in Crowdsourcing

Award: CrowdCO-OP: Sharing Risks and Rewards in Crowdsourcing

Dr Alessandro Checco

A joint work between the University of Queensland, the University of Hanover, and the University of Sheffield titled CrowdCO-OP: Sharing Risks and Rewards in Crowdsourcing has received an Honorable Mention Award at the prestigious Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2020).

The work focused on paid micro-task crowdsourcing. This type of labour has gained in popularity mainly because of the increasing need for large-scale manually labelled datasets which are often used to train and evaluate Artificial Intelligence systems. Modern paid crowdsourcing platforms use a piecework approach to rewards, meaning that workers are paid for each task they complete, given that their work quality is considered sufficient by the requester or the platform. Such an approach creates risks for workers: their work may be rejected without being rewarded, and they may be working on poorly rewarded tasks, in light of the disproportionate time required to complete them. As a result, recent research has shown that crowd workers may tend to choose specific, simple, and familiar tasks and avoid new requesters to manage these risks.

In this work, we propose a novel crowdsourcing reward mechanism that allows workers to share these risks and achieve a standardized hourly wage equal for all participating workers. Reward-focused workers can thereby take up challenging and complex HITs without bearing the financial risk of not being rewarded for completed work. We experimentally compare different crowd reward schemes and observe their impact on worker performance and satisfaction. Our results show that 1) workers clearly perceive the benefits of the proposed reward scheme, 2) work effectiveness and efficiency are not impacted as compared to those of the piecework scheme, and 3) the presence of slow workers is limited and does not disrupt the proposed cooperation-based approaches.

A pre-print of this work is available at this link.

For any questions feel free to contact Alessandro Checco at a.checco@sheffield.ac.uk.

Labels:

Comments

Post a Comment

Popular posts from this blog

Dr Kate Miltner joins UK Young Academy

We are pleased to announce that Dr Kate Miltner has joined the UK Young Academy, an interdisciplinary network of early-career professionals and researchers working together to tackle pressing global and local challenges and promote lasting change. Dr Miltner is among the 42 emerging leaders from across the UK selected as the newest members of the UK Young Academy. The new members come from a wide range of sectors, with backgrounds in political science, engineering, government, communications and the creative and performing industries, and more. As members of the UK Young Academy, they will have the opportunity to take action on both local and global issues. Through interdisciplinary projects and working across sectors, they will bridge gaps, drive innovation, and develop the solutions needed to address critical challenges – all while advancing their professional development and contributing to a global network of Young Academies focused on achieving positive outcomes. "I'm tru...
Post a Comment
Read more

Dr Paul Reilly & Dr Elisa Serafinelli organised IMPROVER workshop, Paris, 10 January 2017

Dr Paul Reilly and Dr Elisa Serafinelli organised a workshop with colleagues from  Euro-Med Seismological Centre (EMSC)  in Paris on 10th January, as part of the Horizon 2020 funded project IMPROVER. Delegates from organisations such as SNCF and VISOV were invited to give their views on the role of social media in disaster response. Thanks to Grigore Havarneanu and the International Union of Railways (UIC) for hosting, and for Laura Petersen and Laure Fallou for their facilitation of the focus group. 
Post a Comment
Read more

My Time in Sheffield as a Visiting Researcher - Dr Abdulhalik Pinar

Returning to Sheffield after more than a decade has been a mixture of nostalgia and new opportunities. I first came here in 2011 to complete my MA in Librarianship, and now, I find myself back as a visiting scholar at the Information School. My time as a visiting researcher at the University of Sheffield has been truly rewarding. I am an academic staff at Harran University in Turkey supported for this visit by Tubitak (The Scientific and Technological Research Council of Turkey). This visit has been a great opportunity for my academic and professional development. I am conducting research on artificial intelligence within GLAM (Galleries, Libraries, Archives, and Museums) institutions. I have especially valued the support of my supervisor Dr. Andrew Cox who has guided me through this process. He is an extremely helpful, supportive and understanding person. The University of Sheffield has provided me with an excellent working environment, surrounded by helpful staff and dynamic research...
Post a Comment
Read more