Skip to main content

Just how good can academic peer review be? Evidence from the near ideal case of theoretical physics

 Academic research normally goes through a peer review stage before it appears in a journal or book. This usually involves an editor selecting two or more scholars from the field that have relevant expertise and asking them to assess the submission. These reviewers then make comments about various aspects of the submission as well as giving an overall recommendation, such as: accept, ask for minor revisions and then accept, ask for major revisions and then reevaluate, or reject. The purposes of this exercise include filtering out flawed studies or papers with little value and helping the authors to improve their work by correcting errors or suggesting additional perspectives to consider.

In an ideal world, every paper that passes peer review is error-free, clear and makes a valuable contribution to academic knowledge. In practice, however, there is no absolute truth and so reviewers must make judgements about the extent to which each work is high enough quality to be published. In theory, reviewers should always agree in their evaluations because they are all experts but in practice disagreement is common.

This is a problem for the safety of the academic record because any disagreement between experts suggests that (a) some substandard research gets published because even though most experts would consider it to be low quality, by chance the two or three reviewers selected liked it, and (b) future authors that evaluate the published research and potentially rely on it for their studies, like reviewers, may not be able to effectively evaluate its quality. Thus, reviewer disagreement points to the flawed nature of the academic publishing system, undermining the validity of academic research.

There are some flaws in the above argument, one of which was addressed in our study. This flaw is that academic research often draws on a range of methodological and theoretical expertise. Thus, reviewers may be chosen for non-overlapping knowledge and may make their recommendations based on different aspects of submissions. This should be particularly the case for complex multi-method research and least likely for narrower studies. In our paper (which itself had to go through peer review), we started from the premise that theoretical physics was a field in which a high degree of reviewer agreement could be expected because it is mathematical and theoretical and therefore does not require specialist knowledge about equipment, processing methods or practical application contexts. Instead, it seems that any theoretical physicist should have a reasonable chance of fully comprehending journal articles on the topic, and would therefore be able to make an overall judgement about submissions. We therefore assessed agreement between reviewers for theoretical physics articles.

We investigated deeper than the overall agreement rate and into different aspects of research quality. The three core dimensions of research quality are usually agreed to be rigour, originality, and significance (to scholarship and/or society) and so we assessed the extent to which reviewers of theoretical physics papers gave the same scores for each of these dimensions. The data came from the SciPost Physics online journal, chosen for being one of the few journals that publishes rigour, originality, and significance scores from reviewers. The results showed that reviewers agreed 40% to 48% of the time for each of the three facets. When they disagreed, it was usually by one point (39% to 46% of all judgements), so larger disagreements were rare. Nevertheless, the results still represent a majority disagreement between reviewers in this nearly perfect case, showing that disagreement between reviewers is a fundamental part of science rather than a by-product of different types of expertise (Figure 1).

- Prof Mike Thelwall

Professor of Data Science

Janusz A. Hołyst, Center of Physics in Economics and Social Sciences, Warsaw University of Technology

Thelwall, M. & Holyst, J. (in press). Can journal reviewers dependably assess rigour, significance, and originality in theoretical papers? Evidence from physics. Research Evaluation. https://doi.org/10.1093/reseval/rvad018

Figure 1. Model of key factors influencing peer review judgements (Figure 10 in: Thelwall & Holyst, in press).

Comments

Popular posts from this blog

Raspberry Pi Weather Project now live

A project to create a raspberry pi weather station is currently live in the Information School.  The Sheffield Pi weather station has been created by Romilly Close, undergraduate Aerospace Engineering student at the University of Sheffield.  The project was funded by the Sheffield Undergraduate Research Experience (SURE) scheme and is being supervised by Dr Jo Bates, Paula Goodale and Fred Sonnenwald from the Information School. Information about the Sheffield Pi station and how to create your own can be found on the project website .  You can also see live data from the Sheffield Pi station on Plot.ly , and further information can also be found on the Met Office Weather Observations Website .    This work compliments the School’s existing project entitled ‘The Secret Life of a Weather Datum’ which explores socio-cultural influences on weather data.  This project is funded under the AHRC’s Digital Transformations Big Data call.  It aims to pilot a new approach to im

Our Chemoinformatics Group wins Jason Farradane Award

The Information School's Chemoinformatics Research Group has been awarded the 2012 UKeiG Jason Farradane Award , in recognition of its outstanding 40 year contribution to the information field. The prize is awarded to the three current members of the group,  Professor Val Gillet , Dr John Holliday and Professor Peter Willett . The judges recognised the Group's status as one of the world's leading centres of chemoinformatics research, a major contributor to the field of information science, and an exemplar in raising the profile of the information profession. The School has a long association with the Farradane prize. Its second recipient was long time member of staff Professor Mike Lynch in 1980.

Professor Mike Thelwall gives inaugural lecture

Professor of Data Science Mike Thelwall recently gave his inaugural lecture at the University of Sheffield, entitled  How helpful are AI and bibliometrics for assessing the quality of academic research? The lecture, delivered in the University's Diamond building, was introduced by Head of the Information School Professor Briony Birdi. It covered Mike's research into whether Artificial Intelligence can inform - or replace - expert peer review in the journal article publication process and what this could look like, as well as to what extent bibliometrics and citation statistics can play a role in assessing the quality of a piece of research. Mike also discussed whether tools like ChatGPT can accurately detect research quality. The inaugural lecture was well attended by colleagues from around the University.