Skip to main content

Award: CrowdCO-OP: Sharing Risks and Rewards in Crowdsourcing

Award: CrowdCO-OP: Sharing Risks and Rewards in Crowdsourcing

Dr Alessandro Checco

A joint work between the University of Queensland, the University of Hanover, and the University of Sheffield titled CrowdCO-OP: Sharing Risks and Rewards in Crowdsourcing has received an Honorable Mention Award at the prestigious Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2020).

The work focused on paid micro-task crowdsourcing. This type of labour has gained in popularity mainly because of the increasing need for large-scale manually labelled datasets which are often used to train and evaluate Artificial Intelligence systems. Modern paid crowdsourcing platforms use a piecework approach to rewards, meaning that workers are paid for each task they complete, given that their work quality is considered sufficient by the requester or the platform. Such an approach creates risks for workers: their work may be rejected without being rewarded, and they may be working on poorly rewarded tasks, in light of the disproportionate time required to complete them. As a result, recent research has shown that crowd workers may tend to choose specific, simple, and familiar tasks and avoid new requesters to manage these risks.

In this work, we propose a novel crowdsourcing reward mechanism that allows workers to share these risks and achieve a standardized hourly wage equal for all participating workers. Reward-focused workers can thereby take up challenging and complex HITs without bearing the financial risk of not being rewarded for completed work. We experimentally compare different crowd reward schemes and observe their impact on worker performance and satisfaction. Our results show that 1) workers clearly perceive the benefits of the proposed reward scheme, 2) work effectiveness and efficiency are not impacted as compared to those of the piecework scheme, and 3) the presence of slow workers is limited and does not disrupt the proposed cooperation-based approaches.

A pre-print of this work is available at this link.

For any questions feel free to contact Alessandro Checco at a.checco@sheffield.ac.uk.

Labels:

Comments

Post a Comment

Popular posts from this blog

Raspberry Pi Weather Project now live

A project to create a raspberry pi weather station is currently live in the Information School.  The Sheffield Pi weather station has been created by Romilly Close, undergraduate Aerospace Engineering student at the University of Sheffield.  The project was funded by the Sheffield Undergraduate Research Experience (SURE) scheme and is being supervised by Dr Jo Bates, Paula Goodale and Fred Sonnenwald from the Information School. Information about the Sheffield Pi station and how to create your own can be found on the project website .  You can also see live data from the Sheffield Pi station on Plot.ly , and further information can also be found on the Met Office Weather Observations Website .    This work compliments the School’s existing project entitled ‘The Secret Life of a Weather Datum’ which explores socio-cultural influences on weather data.  This project is funded under the AHRC’s Digital Transformations Big Data call.  It aims to pilot a new approach to im
Post a Comment
Read more

Our Chemoinformatics Group wins Jason Farradane Award

The Information School's Chemoinformatics Research Group has been awarded the 2012 UKeiG Jason Farradane Award , in recognition of its outstanding 40 year contribution to the information field. The prize is awarded to the three current members of the group,  Professor Val Gillet , Dr John Holliday and Professor Peter Willett . The judges recognised the Group's status as one of the world's leading centres of chemoinformatics research, a major contributor to the field of information science, and an exemplar in raising the profile of the information profession. The School has a long association with the Farradane prize. Its second recipient was long time member of staff Professor Mike Lynch in 1980.
Post a Comment
Read more

Professor Mike Thelwall gives inaugural lecture

Professor of Data Science Mike Thelwall recently gave his inaugural lecture at the University of Sheffield, entitled  How helpful are AI and bibliometrics for assessing the quality of academic research? The lecture, delivered in the University's Diamond building, was introduced by Head of the Information School Professor Briony Birdi. It covered Mike's research into whether Artificial Intelligence can inform - or replace - expert peer review in the journal article publication process and what this could look like, as well as to what extent bibliometrics and citation statistics can play a role in assessing the quality of a piece of research. Mike also discussed whether tools like ChatGPT can accurately detect research quality. The inaugural lecture was well attended by colleagues from around the University.
Post a Comment
Read more