A couple of weeks ago I travelled to Atlanta, USA to attend ACM
FAT* - an interdisciplinary conference that addresses issues of Fairness,
Accountability and Transparency in Machine Learning. Officially, I was there on
the hunt for potential papers and authors to invite to submit their work to Online
Information Review. However, the FAT* field is also closely related
to my research interests around the politics of data and algorithms, and my
teaching on the Information School’s MSc Data Science.
I was keen to check out what was happening in the FAT* community, and feed my
findings back into my teaching and into two new projects I am working on in
this field: CYCAT
& supervising a new PhD student – Ruth Beresford – whose
research will investigate algorithmic bias in collaboration with the Department
for Work and Pensions.
I was privileged to hear a number of great papers – the best
of which engaged critically with issues of social context and justice. My two
favourite papers which I highly recommend for anyone interested in these topics
were:
- Fairness and Abstraction in Sociotechnical Systems by Andrew Selbst, danah boyd, Sorelle Friedler, Suresh Venkatasubramanian and Janet Vertasi; and,
- Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments by Ben Green and Yiling Chen
Fairness
and Abstraction in Sociotechnical Systems (Selbst et al, 2019) was
a great paper that spoke to a sense of unease I have felt recently about the
increasing amount of technical work attempting to solve the ‘algorithmic bias
problem’. The paper – written by an interdisciplinary team of social and
computer scientists - not only speaks to such concerns in an engaging and
insightful way, but also offers a strong analytical framework that illuminates
five “traps” that such technically-driven work often falls into:
- Framing: The authors begin by critiquing the ‘algorithmic framing’ common in data science. In such an abstraction, the focus of the data/computer scientist is simply on evaluating, for example, whether the model has high accuracy. They point out that such a framing is ineffective for addressing issues of bias and fairness. Expanding this algorithmic framing to a ‘data frame’ which also involves interrogating directly the data inputs and outputs for issues of bias and fairness, can address some of these issues, but also has its limitations. Instead they advocate data scientists adopting a socio-technical framing which explicitly recognises that any ML model is part of a socio-technical system – and we need to move all the decisions made by humans and human institutions into the abstraction boundary. I couldn’t agree more!
- Formalism: This is an important ‘trap’ for computer scientists and mathematicians to be aware of. It relates to the failure to account for the complexity of social concepts such as fairness, bias etc. The meaning of such concepts is contextual and contestable – they cannot be reduced to mathematical formalisms!
- The Ripple Effect: This ‘trap’ points to the lack of awareness that when technical solutions are embedded into existing social systems, they can impact upon the behaviours and values of those in the social system – often in unexpected ways.
- Portability: This ‘trap’ relates to the problems inherent in repurposing algorithmic solutions from one social context to another – and the inevitable problems of inaccuracy, misleading results, and potential for harm.
- Solutionism: And, finally - the simple observation that technologists often fail to recognise that the best solution to a problem may not involve any technology!
Putting some of these ideas into practice was my second
favourite paper – which also won the prize for best technical and
interdisciplinary paper - Disparate Interactions: An
Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments (Green and Chen, 2019).
With a focus on risk assessments being used in the US
criminal justice system, the authors argue that given risk assessment tools do
not actually make decisions, but are used to inform judges’ decisions, it is
important to understand how people actually interpret and use the outputs of
these tools. Their study is based on an experiment involving Amazon Mechanical
Turk workers – rather than actual judges – however, their findings are
concerning. They found their participants under-performed the risk assessment
tool even when presented with the prediction of the tool; they were unable to
effectively evaluate the accuracy of their own decisions or those of the tool;
and, most concerning they exhibited biased interaction with the tool’s
prediction whereby use of risk
assessments in decision making led to participants making higher risk
predictions for black defendants and lower risk predictions for white
defendants. Clearly, these
findings need examining ‘in the wild’, but they are concerning, and evidence
the importance of a socio-technical framing as called for by Selbst et al.
While there were some excellent papers presented at FAT*,
there were also a good few that fell into some of the ‘traps’ of abstraction
identified in Selbst et al’s paper – and this resulted in some interesting
commentary about the nature and direction of the field. For example, important
questions were raised by Stanford PhD student Pratyusha Ria Kalluri, who drew
upon Catherine D'Ignazio
and Lauren Klein’s
forthcoming book Data Feminism, to question the language of fairness,
accountability and transparency, and how it relates to notions of justice. Her
comments received a lot of support from attendees and online:
Pratyusha’s observations reflect many of my own – and others
in the Critical Data Studies space - concerns about what it means to work across
disciplinary boundaries in this field, and the politics of engaging in such
work with people who may have very different agendas, assumptions, and
understandings about what is at stake. It can sometimes be difficult to know
how best to navigate these tensions – but it feels like 2019 could be an important
moment for shaping the direction of the field.
