Skip to main content

Award: CrowdCO-OP: Sharing Risks and Rewards in Crowdsourcing

Award: CrowdCO-OP: Sharing Risks and Rewards in Crowdsourcing

Dr Alessandro Checco

A joint work between the University of Queensland, the University of Hanover, and the University of Sheffield titled CrowdCO-OP: Sharing Risks and Rewards in Crowdsourcing has received an Honorable Mention Award at the prestigious Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2020).

The work focused on paid micro-task crowdsourcing. This type of labour has gained in popularity mainly because of the increasing need for large-scale manually labelled datasets which are often used to train and evaluate Artificial Intelligence systems. Modern paid crowdsourcing platforms use a piecework approach to rewards, meaning that workers are paid for each task they complete, given that their work quality is considered sufficient by the requester or the platform. Such an approach creates risks for workers: their work may be rejected without being rewarded, and they may be working on poorly rewarded tasks, in light of the disproportionate time required to complete them. As a result, recent research has shown that crowd workers may tend to choose specific, simple, and familiar tasks and avoid new requesters to manage these risks.

In this work, we propose a novel crowdsourcing reward mechanism that allows workers to share these risks and achieve a standardized hourly wage equal for all participating workers. Reward-focused workers can thereby take up challenging and complex HITs without bearing the financial risk of not being rewarded for completed work. We experimentally compare different crowd reward schemes and observe their impact on worker performance and satisfaction. Our results show that 1) workers clearly perceive the benefits of the proposed reward scheme, 2) work effectiveness and efficiency are not impacted as compared to those of the piecework scheme, and 3) the presence of slow workers is limited and does not disrupt the proposed cooperation-based approaches.

A pre-print of this work is available at this link.

For any questions feel free to contact Alessandro Checco at a.checco@sheffield.ac.uk.

Labels:

Comments

Post a Comment

Popular posts from this blog

My Time in Sheffield as a Visiting Researcher - Dr Abdulhalik Pinar

Returning to Sheffield after more than a decade has been a mixture of nostalgia and new opportunities. I first came here in 2011 to complete my MA in Librarianship, and now, I find myself back as a visiting scholar at the Information School. My time as a visiting researcher at the University of Sheffield has been truly rewarding. I am an academic staff at Harran University in Turkey supported for this visit by Tubitak (The Scientific and Technological Research Council of Turkey). This visit has been a great opportunity for my academic and professional development. I am conducting research on artificial intelligence within GLAM (Galleries, Libraries, Archives, and Museums) institutions. I have especially valued the support of my supervisor Dr. Andrew Cox who has guided me through this process. He is an extremely helpful, supportive and understanding person. The University of Sheffield has provided me with an excellent working environment, surrounded by helpful staff and dynamic research...
Post a Comment
Read more

Tools for Life: Data Sharing and Public Health

 Data is the lifeblood of our digital society, but how our personal information is stored and shared is an understandably contentious issue. ‘ Tools for Life: Data Sharing and Public Health ’, a 20-month public engagement project which concluded in March 2021 and was headed by Information School Lecturer Dr Jonathan Foster, looked into the issues surrounding this idea in the context of patient data, with consideration of this issue being particularly important against the backdrop of the current NHS opt-out system for consent. Dr Jonathan Foster Alongside Dr Foster, the project involved fellow Information School Lecturer and expert in Health Informatics Dr Laura Sbaffi, and Professor Suzanne Mason, a clinician academic and Professor of Emergency Medicine based at the University’s School of Health and Related Research (ScHARR). With Jonathan’s expertise being in Information Management, the project had a broad knowledge base from both the health and the data side of the equation. The...
Post a Comment
Read more

Generative AI paper authored by Dr Kate Miltner among British Academy's 13 discussion papers on "good" digital society

 The British Academy has today published thirteen discussion papers from a range of expert perspectives across the ‘SHAPE’ disciplines (Social Sciences, Humanities and the Arts for People and the Economy) to explore the question: ‘What are the possibilities of a good digital society?’  The papers explore a wide range of issues, from the environmental impacts of digitalised daily life to the possibilities of ‘good’ Generative AI in the cultural and creative industries, to examining more closely what we mean by a ‘good digital society’. Among the papers is one authored by information School Lecturer Dr Kate Miltner, with Dr Tim Highfield from the Department of Sociological Studies. Their paper focuses on "good" uses of generative AI in the cultural & creative industries. Alongside the papers is an introductory summary that provides a thematic overview of the papers and points to how we might conceptualise the principles that underpin these diverse visions of a good digital ...
Post a Comment
Read more