Press release issued:
4 December 2013
The effectiveness of ‘peer review’ — the system whereby scientists critique each other’s research to determine whether or not it should be published — is investigated in new findings published in Nature.
Researchers from the University of Bristol suggest that the system, which is one of the main ways in which science attempts to ensure quality research, is open to flaws but identify an improved model that would better ensure the quality and veracity of published work.
Using a mathematical model of the behaviour of scientists, the team aimed to better understand how the model of peer review works, and in particular whether it may be susceptible to failure by examining the factors that can influence scientists when critiquing the work of others.
As reviewers are often encouraged to consider only largely objective characteristics of the study they are reviewing, the team found that they are open to "herding" — a phenomenon whereby scientists’ behaviour may be influenced, and even dominated, by information gleaned for their peers’ behaviour rather than by their own personal opinions. The researchers found that this increases the risk of reviewers converging on what they perceive to be a correct answer but is actually an incorrect answer.
The study found that peer review performs best when a degree of subjectivity (i.e., their belief about whether the result is correct) is allowed, since this enables more information to be transmitted through the decision and protects against the risk of scientists converging on an incorrect answer.
An improved peer review model includes the opportunity for scientists to comment on and critique research after it has been published. This would provide scientists with more opportunity to truthfully reveal their opinions, and improve the flow of information in science and avoid the risk of herding.
Mike Peacey, one of the study’s researchers who is based jointly at the Universities of Bristol and Bath, said: “Scientists are increasingly concerned that many published research findings may be false. Our findings demonstrate a novel way of how the peer review process can be improved through a post-publication peer review.”
The study was funded by the Economic and Social Research Council (ESRC) and the Medical Research Council (MRC).
Further information
Paper
‘Modelling the effects of subjective and objective decision making in scientific peer review’ by In-Uck Park (University of Bristol and Sungkyunkwan University, South Korea), Mike W Peacey, (Universities of Bristol and Bath) and Marcus Munafò (University of Bristol) is published in the journal Nature.
About the peer review process
The peer review process for research papers submitted to scientific journals comprises several stages. Once the study’s lead author has written up the results of their study, they submit their paper to a relevant journal. Once the paper passes the journal’s initial quality-control process it’s assigned an academic editor with expertise related to the paper. It is this editor who decides whether the paper is appropriate for the journal and category of article that it was submitted to. If the paper passes this stage it is then sent out for critiquing by experts in the field. If it doesn’t pass then the paper is returned to the author who will submit to a more appropriate journal.
Once the reviewers accept the invitation to review a manuscript, they are then expected to submit a completed report as to the significance, novelty and scientific rigor of the paper. Most editors ask the reviewers to rank or grade the significance of the work, the novelty of the research as well as experimental design or approach. In addition to providing these grades or ranks, the reviewers also offer specific comments about the strengths and weaknesses of a paper.
At the final stage reviewers provide the editors with an overall recommendation, such as reject the paper, return the paper to authors for major revisions, return paper to authors for minor revisions or accept the paper as is. The editor then receives the reviewer reports and decides upon the overall verdict on the paper. If the reviews are too disparate, the editor may seek the opinion of additional peer reviewers to help with a final decision. The final decision letter contains editorial comments as well as the comments provided by the reviewers to explain why a certain decision was reached.
