Information School News

The Information School, in collaboration with the University of Rome "Tor Vergata", has published the paper " AI-assisted peer review " in the Nature journal Humanities and Social Sciences Communications, by Alessandro Checco, Lorenzo Bracciale, Pierpaolo Loreti, Stephen Pinfield and Giuseppe Bianchi.  Image via https://www.vpnsrus.com/ In the context of academic research, we designed an experiment to test AI capabilities in predicting the review score of manuscripts. We show that such techniques can reveal correlations between the decision process and other quality proxy measures, uncovering potential biases of the review process. We discuss the opportunities, but also the potential unintended consequences of these techniques in terms of algorithmic bias and ethical concerns.

BHCC 2020 - 2nd Symposium on Biases in Human Computation and Crowdsourcing Dr Alessandro Checco Human Computation and Crowdsourcing have become ubiquitous in the world of algorithm augmentation and data management. However, humans have various cognitive biases that influence the way they make decisions, remember information, and interact with machines. It is thus important to identify human biases and analyse their effect on complex hybrid systems. On the other hand, the potential interaction with a large pool of human contributors gives the opportunity to detect and handle biases in existing data and systems. The goal of this symposium is to analyse both existing human biases in hybrid systems, and methods to manage bias via crowdsourcing and human computation. We will discuss different types of biases, measures and methods to track bias, as well as methodologies to prevent and solve bias. An interdisciplinary approach is often required to capture the broad effects that these processe...

Award: CrowdCO-OP: Sharing Risks and Rewards in Crowdsourcing Dr Alessandro Checco A joint work between the University of Queensland, the University of Hanover, and the University of Sheffield titled CrowdCO-OP: Sharing Risks and Rewards in Crowdsourcing has received an Honorable Mention Award at the prestigious Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2020). The work focused on paid micro-task crowdsourcing. This type of labour has gained in popularity mainly because of the increasing need for large-scale manually labelled datasets which are often used to train and evaluate Artificial Intelligence systems. Modern paid crowdsourcing platforms use a piecework approach to rewards, meaning that workers are paid for each task they complete, given that their work quality is considered sufficient by the requester or the platform. Such an approach creates risks for workers: their work may be rejected without being rewarded, and they may be working on poorly...

From Director of Research Professor Paul Clough: I am delighted to announce that Alessandro Checco and Jo Bates (together with Gianluca Demartini) have won the Best Paper award at the prestigious Human Computation or HCOMP 2018 conference for the following paper: Checco A, Bates J & Demartini G (2018) All That Glitters is Gold -- An Attack Scheme on Gold Questions in Crowdsourcing. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. Abstract here; http://eprints.whiterose.ac.uk/130654/ Not only is it a significant achievement to even be accepted at this conference it is an outstanding achievement to be nominated for Best Paper and then to win it is incredible. Alessandro and Gianluca were awarded the prize at HComp 2018 . Alessandro had this to say about the paper and reviews: "Feedback from chairs was that they really liked the fact we opened a new direction (that is having workers using ML solutions on the employers). We will have the opportu...