On 20 May 2015 Dr Gianluca Demartini of the Information School will present a paper on 'The Dynamics of Micro-Task Crowdsourcing' at the 24th World Wide Web Conference in Florence, Italy.
Micro-task crowdsourcing is a modern technique that allows outsourcing of simple data collection tasks to a crowd of individuals online. Tasks such as image annotation, document summarisation, or audio transcription are easy for humans to complete but very challenging for computers. micro-task crowdsourcing is commonly used to build information systems that combine the scalability of computers over large amounts of data with the quality of human intelligence.
Over the last 10 years different micro-task crowdsourcing platforms have been created. These platforms are marketplaces where crowd workers complete tasks (usually called Human Intelligence Tasks or HITs) in exchange of small monetary rewards and where requesters post their data and tasks to quickly obtain large scale annotations.
As part of research carried out with Difallah, Catasta, Ipeirotis and Cudré-Mauroux, Gianluca analysed logs between 2009 and 2014 from the most popular crowdsourcing platform: Amazon Mechanical Turk and observed the evolution over time of its usage. This research will be presented at the World Wide Web Conference, and the full paper is available here.
The main findings from the research are:
Published tasks:
- The most frequent HIT reward value on Amazon Mechanical Turk has increased over time, and reaches $0.05 in 2014.
- HITs about audio transcription have been gaining momentum over last years and are now the most popular tasks on Amazon Mechanical Turk.
- Content Access HITs (like “Visit this website” or “click this link”) popularity on Amazon Mechanical Turk has decreased over time.
- Surveys are the most popular type of HITs for US-based workers on Amazon Mechanical Turk.
Workers and Requesters:
- HITs on Amazon Mechanical Turk that are exclusively asking for workers based in India have strongly decreased over time
- While most HITs on Amazon Mechanical Turk do not require country-specific workers, most of such HITs require US-based workers
- New requesters constantly join Amazon Mechanical Turk, making the number of active requesters and available reward increase over time: Over the last 2 years, an average of 1000 new requesters per month joined Amazon Mechanical Turk
- There is a weekly seasonality effect in the amount of rewards assigned to workers and in the HITs available on Amazon Mechanical Turk
Market size and dynamics:
- On Amazon Mechanical Turk 10K new HITs arrive and 7.5K HITs get completed every hour (on average)
- New HITs attract new workers to the Amazon Mechanical Turk website
- New workers arriving to the Amazon Mechanical Turk platform complete both fresh and old HITs
- Workers on Amazon Mechanical Turk prefer to work on fresh, recently posted HITs
- New work has almost 10x higher attractiveness for workers as compared to remaining work on Amazon Mechanical Turk
Work size and speed:
- Very large (300K HITs) batches recently appeared on Amazon Mechanical Turk
- Throughput of HIT batches on Amazon Mechanical Turk can best be predicted based on the number of HITs in the batch and its freshness
- Large HIT batches can achieve high throughput (thousands of HITs per minute) on Amazon Mechanical Turk
Micro-task crowdsourcing is a modern technique that allows outsourcing of simple data collection tasks to a crowd of individuals online. Tasks such as image annotation, document summarisation, or audio transcription are easy for humans to complete but very challenging for computers. micro-task crowdsourcing is commonly used to build information systems that combine the scalability of computers over large amounts of data with the quality of human intelligence.
Over the last 10 years different micro-task crowdsourcing platforms have been created. These platforms are marketplaces where crowd workers complete tasks (usually called Human Intelligence Tasks or HITs) in exchange of small monetary rewards and where requesters post their data and tasks to quickly obtain large scale annotations.
As part of research carried out with Difallah, Catasta, Ipeirotis and Cudré-Mauroux, Gianluca analysed logs between 2009 and 2014 from the most popular crowdsourcing platform: Amazon Mechanical Turk and observed the evolution over time of its usage. This research will be presented at the World Wide Web Conference, and the full paper is available here.
The main findings from the research are:
Published tasks:
- The most frequent HIT reward value on Amazon Mechanical Turk has increased over time, and reaches $0.05 in 2014.
- HITs about audio transcription have been gaining momentum over last years and are now the most popular tasks on Amazon Mechanical Turk.
- Content Access HITs (like “Visit this website” or “click this link”) popularity on Amazon Mechanical Turk has decreased over time.
- Surveys are the most popular type of HITs for US-based workers on Amazon Mechanical Turk.
Workers and Requesters:
- HITs on Amazon Mechanical Turk that are exclusively asking for workers based in India have strongly decreased over time
- While most HITs on Amazon Mechanical Turk do not require country-specific workers, most of such HITs require US-based workers
- New requesters constantly join Amazon Mechanical Turk, making the number of active requesters and available reward increase over time: Over the last 2 years, an average of 1000 new requesters per month joined Amazon Mechanical Turk
- There is a weekly seasonality effect in the amount of rewards assigned to workers and in the HITs available on Amazon Mechanical Turk
Market size and dynamics:
- On Amazon Mechanical Turk 10K new HITs arrive and 7.5K HITs get completed every hour (on average)
- New HITs attract new workers to the Amazon Mechanical Turk website
- New workers arriving to the Amazon Mechanical Turk platform complete both fresh and old HITs
- Workers on Amazon Mechanical Turk prefer to work on fresh, recently posted HITs
- New work has almost 10x higher attractiveness for workers as compared to remaining work on Amazon Mechanical Turk
Work size and speed:
- Very large (300K HITs) batches recently appeared on Amazon Mechanical Turk
- Throughput of HIT batches on Amazon Mechanical Turk can best be predicted based on the number of HITs in the batch and its freshness
- Large HIT batches can achieve high throughput (thousands of HITs per minute) on Amazon Mechanical Turk
Above: Cumulative HITs (log) per country plotted by time
Above: Micro Reward per year
Comments