NEW PERSPECTIVES ON PEER REVIEW
- Case Study: The Canadian Journal of History/Annales canadiennes d’histoire
- What’s the problem with Peer Review?
Peer review is a system of evaluation employed in academic publishing conducted by expert scholars to evaluate the quality and veracity of academic research. At a peer reviewed publication, a group of editors evaluates submitted articles and selects pieces that match their publication criteria for further. These materials are sent to other scholars, considered experts in their respective sub-specialities, to advise the editors about whether the research is factual, well-argued, and innovative in their respective fields. Although peer review relies on individuals — usually external — as expert commentators, articles are chosen for review, decisions are ultimately made based on reviewer commentary, and the invitations to reviewers themselves are sent by a publication’s editors. Having a thoughtful set of editorial policies and practices around selection and peer review is an essential way to ensure that what you publish matches your publication’s mission and serves your community.
It is important to remember that the entire peer review system revolves around opinions from individuals and is, therefore, highly subjective. This can lead to bad practice, both actions in bad faith or simply decision making that is influenced by inherent biases or perpetuates outdated systems. Many of the criticisms leveled against peer review – about fairness, diversity, opportunity – lead us to ask the questions: Who is choosing what we publish? and How do these people and decisions impact our scholarship?
What’s the Problem with Peer Review?
Peer review is for quality assurance and fact checking, and arguably help to shape and improve scholarship by helping authors to improve their writing and the structure of their arguments.
‘Blind Review’, and that can be “Single Blind,” where the reviewer knows the identity of the author or, more frequently, Double Blind, where both the author and referee are anonymous, has been considered a system where research can be rigorously vetted by experts and where anonymity provides the basis for objective, unbiased evaluation.
However, everyone operates under their own inherent biases. They might be critical of certain arguments, methodologies, and subjects on the basis of gender, language, nationality/race. They might also be subject to confirmation bias – adhering to conservative or widely held beliefs, or rejecting out of hand rather than engaging with and valuing innovation in process and argument. Even in blind review, with a lack of diversity in peer reviewers and their perspectives, bias can be a real issue for authors with interests in emerging fields or non-mainstream thinking. This system of anonymity can also create a lack of accountability, unscrupulous editors can ignore or obscure the results of peer review, reviewers can provide poor or even mean spirited commentary, knowing authors won’t see their exact comments or won’t know who made them.
Many people question the actual value and quality of review. As a subjective system it is prone to human error and unreliability — faulty research and research findings can get through this process where only one or two individuals have been relied on. Peer reviewers in blind systems don’t have a lot of incentive to craft thoughtful, thorough and timely review. Just getting peer reviewers to accept invitations and agree to give their time and energy to your publication can be a difficult sell, after all they don’t receive any credit for the work.
How do we make it better?
- Form diverse editorial boards
- Create affirmative policies around inviting diverse reviewers, editors, and other contributors (like book reviewers)
- Use process of “triple-blind” review to anonymize information about submitters where articles and abstracts are reviewed without information that would identify gender, nationality, institutional affiliation, etc.
- This makes it more difficult for editors to identify the “qualified” or “eminent” scholars, and creates opportunities for emerging scholars
- Try an “Open” peer review policy
Open Peer Review is a malleable term that generally describes a system in which the identity of participants is not disguised, but can take on a variety of forms. Some publications that practice open review follow a fairly traditional format of editor-mediated review, where the participants know each other’s names. However, open review has been practiced in a lot of interesting ways and has really been tessellated and shaped in experimental ways.
Tony Ross-Hellaur, a librarian and information science researcher, has attempted to create a system of classification for open peer review developing categories, that can help us understand the wide variety of open peer review practices that are being used today. Some of these are discussed and illustrated in the presentation and include:
- Open identities: Authors and reviewers are aware of each other’s identity
- Open reports: Review reports are published alongside the relevant article.
- Open participation: The wider community are able to contribute to the review process.
- Open interaction: Direct reciprocal discussion between author(s) and reviewers, and/or between reviewers, is allowed and encouraged.
- Open pre-review manuscripts: Manuscripts are made immediately available (e.g., via pre-print servers like arXiv) in advance of any formal peer review procedures.
- Open final-version commenting: Review or commenting on final “version of record” publications.
- Open platforms (“decoupled review”): Review is facilitated by a different organizational entity than the venue of publication.
Source: Ross-Hellauer T. What is open peer review? A systematic review [version 2; referees: 4 approved]. F1000Research 2017, 6:588 (doi: 10.12688/f1000research.11369.2)
At its best, peer review can and should be an opportunity for scholarly conversation, where authors can improve and refine their research and writing, reviewers feel they are doing a valuable service to their academic communities and improve their own communication and assessment skills. Open peer review can help to combat inherent bias and improve reviewer feedback because it increases accountability. And it motivates peer reviewers to participate and to do so in a robust way by giving them public credit for their time and work.
When peer review commentary and results are shared publicly, readers can even benefit from the opportunity to see counterpoints and arguments, so that they have a more nuanced or critical view on the subject, by seeing critique and suggestion from another field expert, in counterpoint or conversation with the author. Anyone can also learn how to be a better reviewer by reading thoughtful critique and seeing how it can help authors to shape and refine their texts. How are students, the scholars of tomorrow, supposed to learn about peer review, a system that they as scholars will engage in on both sides, if it is happening entirely in a “black box”?
Case Study: The Canadian Journal of History/Annales canadiennes d’histoire
The editorial board select works that will be advanced to peer review as well as the reviewers who will carry out those assessments. A case study of the Canadian Journal of History based on a 2018 article by Rilla Friesen can help us to think about what kind of actions and policies we can implement to meet goals of ethical, meaningful review.
The journal is produced by the history department at the University of Saskatchewan and was founded in 1966. Their editorial leadership is composed of editors drawn from of the University’s history department. Like many academic journals, the Canadian Journal of History doesn’t commission its articles, rather authors submit their research for consideration through unsolicited general submissions or in response to calls for papers for the themed issues. Although the journal was following a rigorous system of double-blind peer review (where an author’s text is reviewed anonymously by one or more peer referees and comments are anonymously conferred back to the author by an editor), they were still experiencing a high gender bias toward male authors. Although the average membership of the Canadian Historical Association was around 54% male/46% female/>1% transgender, the journal’s authorship was well below a 60/40 split. The journal set a goal to increase its authorship to an even 50/50 gender split.
As is reflected in Friesen’s graphs, the journal still receives a higher volume of unsolicited articles from male writers, and they still accept more articles written by men than women.
However, in selecting and inviting peer reviewers, the journal was able to take a more active approach, instating a policy to always invite a female reviewer first. The enacting of this policy in 2017 resulted in a nearly equal gender distribution of reviewers, as opposed to 2016, in which only 18% of all reviewers were female. When the invited female reviewer declined the invitation, the editors tended to send the next invitation to a male scholar, underlining the need for intentionality in referee selection to counteract inherent bias toward male colleagues and scholars
Although they do not commission articles, the journal does invite hundreds of scholars to contribute book reviews to the journal, publishing 40-50 reviews each issue. The editors first implemented a policy to invite a woman first, and then later, to issue the first 2 invitations to women. Table 3 shows how once the policy to issue two invitations to women was put into place (by issues 5, the number of women reviewers reached 46%, brining the journal nearly to its goal of parity in authorship.
The Canadian Journal of History editors made leadership choices, enacting policies that created tangible changes in the community of scholars encapsulated in their publication. The CJH was focused on gender parity – an initiative that focused on correcting historic and unethical inequities in publishing and academia and on improving the quality of their publication. We should always emphasize that diversity initiatives advance the mission of our publications and the very goal of the peer review process: to generate the best possible scholarship.
Scholarship comes from people – from authors and the people in the publishing process who select and edit their writing. Increasing diversity in scholarship means making space for oppressed and elided voices, people who will speak about and amplify different subjects and speak from new perspectives. The peer review process, the way that we choose and vet what is published into the scholarly record, is integral in the process of evaluating, questioning, and strengthening what we publish, but it can and should also help to improve diversity in scholarship through inclusive action. As your journal will publish new voices, new topics, more innovative thinking and methodologies, it can become a more unique and necessary contributor to the scholarly record.
Creating Excellent Review through Editorial Practice
As editors we make the decisions about our peer review policies and we also write and present those policies and our thoughts behind them to the public. A policy for open peer review that frames the practice as an opportunity for collegial conversation and to support fellow scholars in their research and writing may win over skeptics of a new system. Being clear in your guidelines and in the forms and questions that reviewers respond to directly is how you get the kind of engagement and feedback that you want to present to your authors.
- Provide clear guidelines and forms that structure reviewers’ feedback
- Create a peer review form with questions that guide referees and provoke meaningful responses.
- Avoid yes or no questions
- Give recognition to reviewers, either through open peer review, or a system like Publons, to show appreciation and incentive participation and real engagement with the process.
At this point in our workshop we took some time to consider and critique three very different examples of peer review policy and guidelines from Nature, Ada, and the Public Philosophy Journal. (You can take a look at and download the materials we used in the workshop by clicking these links). Noting that each of these policies was highly tailored to the field, content, and personality of these publications, we noted how important tone and language could be in creating an attitude in the referee toward the review process. We talked about how length and specificity of the requirements could be overwhelming. We took away the lesson that policy and guidelines are an opportunity to pass a message and ethos to your reviewers about their participation in your publication and can work in concert with your review form, which can provide more structure to written reviews.
For more guidance on writing your own materials, visit Peer Review Guidelines and Forms and Sample Peer Review Guidelines in the Editorial Workbook.
You can also find examples of Peer Review Policies and Guidelines under Tools and Templates.
Sources:
Source: Ross-Hellauer T. What is open peer review? A systematic review [version 2; referees: 4 approved]. F1000Research 2017, 6:588 (doi: 10.12688/f1000research.11369.2)
Friesen, Rilla. (2018). The Past and Present of the Canadian Journal of History / Annales canadiennes d’histoire. Scholarly and Research Communication, 9(1): 0101275, 7 pp. https://src-online.ca/index.php/src/article/download/275/543/Cached
