Information School News

The Information School, in collaboration with the University of Rome "Tor Vergata", has published the paper " AI-assisted peer review " in the Nature journal Humanities and Social Sciences Communications, by Alessandro Checco, Lorenzo Bracciale, Pierpaolo Loreti, Stephen Pinfield and Giuseppe Bianchi.  Image via https://www.vpnsrus.com/ In the context of academic research, we designed an experiment to test AI capabilities in predicting the review score of manuscripts. We show that such techniques can reveal correlations between the decision process and other quality proxy measures, uncovering potential biases of the review process. We discuss the opportunities, but also the potential unintended consequences of these techniques in terms of algorithmic bias and ethical concerns.

BHCC 2020 - 2nd Symposium on Biases in Human Computation and Crowdsourcing Dr Alessandro Checco Human Computation and Crowdsourcing have become ubiquitous in the world of algorithm augmentation and data management. However, humans have various cognitive biases that influence the way they make decisions, remember information, and interact with machines. It is thus important to identify human biases and analyse their effect on complex hybrid systems. On the other hand, the potential interaction with a large pool of human contributors gives the opportunity to detect and handle biases in existing data and systems. The goal of this symposium is to analyse both existing human biases in hybrid systems, and methods to manage bias via crowdsourcing and human computation. We will discuss different types of biases, measures and methods to track bias, as well as methodologies to prevent and solve bias. An interdisciplinary approach is often required to capture the broad effects that these processe...

Award: CrowdCO-OP: Sharing Risks and Rewards in Crowdsourcing Dr Alessandro Checco A joint work between the University of Queensland, the University of Hanover, and the University of Sheffield titled CrowdCO-OP: Sharing Risks and Rewards in Crowdsourcing has received an Honorable Mention Award at the prestigious Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2020). The work focused on paid micro-task crowdsourcing. This type of labour has gained in popularity mainly because of the increasing need for large-scale manually labelled datasets which are often used to train and evaluate Artificial Intelligence systems. Modern paid crowdsourcing platforms use a piecework approach to rewards, meaning that workers are paid for each task they complete, given that their work quality is considered sufficient by the requester or the platform. Such an approach creates risks for workers: their work may be rejected without being rewarded, and they may be working on poorly...

New Paper: Integrating FATE/Critical Data Studies into Data Science Curricula: Where are we going and how do we get there? A number of Information School academics have published a new paper for the ACM Conference on Fairness, Accountability, and Transparency. The paper (published by Dr Jo Bates, Dr David Cameron, Dr Alessandro Checco, Professor Paul Clough, Dr Frank Hopfgartner, Dr Suvodeep Mazumdar, Dr Laura Sbaffi, Dr Peter Stordy, and Dr Antonio de la Vega de León) reflects on the ethical and practical aspects of teaching critical data science to inform future data practices & make them fair.  Abstract:  There have been multiple calls for integrating topics related to fairness, accountability, transparency, ethics (FATE) and social justice into Data Science curricula, but little exploration of how this might work in practice. This paper presents the findings of a collaborative autoethnography (CAE) engaged in by a MSc Data Science teaching team b...

From Director of Research Professor Paul Clough: I am delighted to announce that Alessandro Checco and Jo Bates (together with Gianluca Demartini) have won the Best Paper award at the prestigious Human Computation or HCOMP 2018 conference for the following paper: Checco A, Bates J & Demartini G (2018) All That Glitters is Gold -- An Attack Scheme on Gold Questions in Crowdsourcing. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. Abstract here; http://eprints.whiterose.ac.uk/130654/ Not only is it a significant achievement to even be accepted at this conference it is an outstanding achievement to be nominated for Best Paper and then to win it is incredible. Alessandro and Gianluca were awarded the prize at HComp 2018 . Alessandro had this to say about the paper and reviews: "Feedback from chairs was that they really liked the fact we opened a new direction (that is having workers using ML solutions on the employers). We will have the opportu...