Different forms of peer review policies

Traditionally, peer review occurs between the submission and publication of a manuscript. However, two new peer review procedures have been proposed. Pre-submission peer review, also called ‘registered reports’ or ‘data-free review’, has been introduced to address the growing concern of publication bias favouring positive research results and is thought to prevent data tweaking (Nosek and Lakens 2014). The format entails peer review of research protocols before empirical research is carried out. Usually, review takes place in two stages, the first only based on research questions, motivation of research and intended methodology, and the second stage occurring after data is gathered and analysed to check whether the final manuscript complies with the research protocol. After the first stage, protocols are either rejected or ‘conditionally accepted’: on condition the final manuscript complies with the initial protocol (Center for Open Science 2018). One of the main advantages of this format is that review is carried out at a moment that changes to the experimental set-up can still be made, which is in contrast to traditional peer review where flaws in experimental design can only be detected post-hoc (Center for Open Science 2018). Our data suggests that pre-submission peer review is associated with fewer retractions, although systematic evidence about the advantages of pre-submission review is still limited (Horbach and Halffman 2019).

Besides pre-submission peer review, post-publication peer review has been introduced to facilitate faster knowledge exchange, including scrutiny and qualification of published research findings by a wider audience. The practice has been in use at pre-print servers for several decades already and is currently adopted by several journals as well (Knoepfler 2015, Walker and Rocha da Silva 2015). While evidence about the effects of these innovations is scarce, it does appear to suggest benefits, in particular for pre-submission review.

There are several aspects of a manuscript that can be taken into account, or deliberately not be taken into account, during the peer review process. The list above contains some of the most commonly used criteria (Horbach and Halffman 2019). Not focusing on the anticipated impact and/or novelty of research findings is thought to decrease publication bias (i.e. publishing only positive results), and practices such as data tweaking and HARKing (PLOS 2018). In fact, a study found that selecting on these criteria is associated with more retractions (Horbach and Halffman 2019). Other studies have shown that reviewers are most likely to comment on aspects of theoretical embedding and framing of a study, rather than its empirical soundness (Strang and Siler 2015). Hence, whatever criteria you believe to be appropriate, it seems important to carefully instruct reviewers about them (Malicki, et al., 2019).

In the vast majority of journals manuscripts are reviewed by referees selected by editors (Horbach and Halffman 2019). Alternatively or additionally, manuscripts may be reviewed by an editor-in-chief, an editorial committee, external reviewers selected or suggested by authors, the wider community/readers, and/or commercial review platforms.

While there are reasons to use reviewers suggested by authors (e.g. to identify a wider pool of reviewers) this entails a considerable risk of fake reviews. Either the suggested reviewer can be a made-up person, or the suggested reviewers are real and appropriate experts, but helpfully suggested email addresses lead back to the author, and thus generate a positive review. To prevent fake review while using author suggested reviewers, check the credibility of the institute name of the suggested reviewer, check if is the email address is provided by a third party (e.g. gmail/hotmail), check for indications of conflict of interest, check the appropriateness of the expertise of the reviewer, and check if the suggested reviewer regularly co-authors with the corresponding author (Gao and Zhou 2017, Tancock 2018).

In addition, the wider community/readers and commercial platforms can also be included as peer reviewers, for instance through post-publication review (Research Square 2017). The main advantage of commercial review platforms is that reviews can be delivered faster and more efficiently, since it reduces the likelihood of a manuscript going through multiple rounds of peer review (Horbach and Halffman 2019). The rationale behind the use of the wider community/readers is that more people will critically look at the paper. Furthermore, involvement of the wider community/readers is a way to broaden and diversify the peer review process which is in line with recent trends towards open science (Funtowicz and Ravetz 2001). A potential drawback of this kind of review is that it has turned out to be difficult to engage sufficient numbers of reviewers for all manuscripts (Campbell 2006, Knoepfler 2015). While wider community review has strong advocates, hard evidence that it traces error more effectively is not yet available.

As part of the current trend towards open science, some also advocate open peer review (Ross-Hellauer, Deppe et al. 2017). However, concerns regarding open peer review are fairness and equality. There are indications that there are biases in peer review as a result of the social position of authors (Ceci and Peters 1982 , Ross, Gross et al. 2006). To prevent (unconscious) biases, and to promote judgement of manuscripts solely on content, authors’ identities can be blinded to reviewers and/or editors. In general, people are stricter when judging the unknown rather than the known or familiar (Cao, Han et al. 2009). This also seems to be the case for peer review, since blinding the author identity to the reviewers is related to fewer retractions (Horbach and Halffman 2019).

In addition to blinding the authors’ identity to reviewers, some journals also blind the authors’ identity to editors, to prevent (unconscious) bias of editors (Tennant, Dugan et al. 2017). However, due care has to be taken in the blinding process, with various strategies available, ranging from only deleting author names from the title page, to omitting self-citations, to obscuring any potential reference to the authors’ including geographical locations of field work and nationality of research participants. It has been argued that no matter the strategy taken, reviewers are usually capable to identify authors nevertheless, hence questioning the effectiveness of blinding (Pontille and Torny 2014). However, this may depend on the size of research fields. Preferences for different forms of author blinding vary strongly among researchers and consensus on this issue is not likely to come soon.

The use of anonymous reviewers is very common, although there are recent calls for open peer review (Ross-Hellauer, Deppe et al. 2017, Horbach and Halffman 2019). “Open peer review” can refer to a wide variety of characteristics of peer review, among which the identity of the reviewers. Traditionally, reviewers’ identities are known only to the editors, but in open review the identities can be disclosed to the authors and the even the readers (Ross-Hellauer 2017). It is argued that open peer review would increase the quality of peer review by leading to more constructive feedback, giving credit to the reviewer, and reducing reviewers’ bias (Godlee 2002). In addition, the disclosure of reviewers’ identities could reduce improper behaviour of reviewers, such as unjustly advising rejection, deliberately delaying publication or plagiarising the manuscript under review (Godlee 2002, Pontille and Torny 2015, Walker and Rocha da Silva 2015, Tennant, Dugan et al. 2017). However, some fear that open peer review could decrease the quality of peer review, since reviewers might fear professional reprisal when submitting critical or negative reviews (Bruce, Chauvin et al. 2016, Ross-Hellauer 2017, Ross-Hellauer, Deppe et al. 2017). This might especially affect early career researchers or others perceiving themselves in vulnerable positions. However, research into this potential fear is rather limited.

Traditionally, access to review reports is restricted to authors and editors, which is the case in the vast majority of journals (Horbach and Halffman 2019). Recently, however, there is a trend towards more openness in science, including the review process. Therefore, some journals publish the signed or anonymous review reports alongside the article. There are several potential advantages to publishing review reports. First, readers can make more informed decisions on the validity of the research and its conclusions (Ceci and Peters 1982). Second, it is thought that reviewers will be more precise and constructive when their review reports will be published. Finally, open reports can serve as a guide for young researchers to help them start with reviewing (Ross-Hellauer 2017).

Some studies have aimed to assess whether changing the accessibility of review reports affects the quality of the review, but the results remain inconclusive. Some studies are rather positive and indicate that publishing the review reports does not influence reviewers’ willingness to review, the turn-around time, and publicized comments are longer and more likely to result in a constructive dialogue (Bornmann, Wolf et al. 2012, Bravo, Grimaldo et al. 2019). On the other hand, another study shows that the possibility of publicly accessible review reports decreases the willingness to review and increases the amount of time taken to write a review report (Van Rooyen, Delamothe et al. 2010). In addition, some have expressed their concerns about the already overburdened publication system, being loaded with additional material for researchers to read in their already limited time.

Traditionally, no interaction between authors/reviewers is facilitated apart from communicating formal review reports and responses to those. However, some journals facilitate additional interaction amongst reviewers, or between authors and reviewers. For instance, some journals have implemented discussions forums in which reviewers can discuss their assessments of manuscripts to come to consensus before submitting their reports to the authors (Schekman, Watt et al. 2013). Other journals merely facilitate reviewers asking each other or the editor questions or expressing concerns about a manuscript under review. In addition, some journals have experimented with forums in which authors and reviewers can (openly) discuss a manuscript under review.

In one study, we found that no interaction between authors/reviewers is associated with fewer retractions (Horbach and Halffman 2019). However, another study is positive towards interaction between reviewers. The vast majority (95%) of reviewers thinks that interaction between reviewers is valuable to authors (King 2017). One of the proposed downsides of interaction between reviewers, is an increase in time required to finalise reviews. However, it is thought to be an efficient investment of time since it could prevent several rounds of miscommunication and re-reviewing. Indeed, in a study on interaction between reviewers over 70% of the articles were accepted after a single round of review (King 2017). Some journals do not only support interaction between reviewers, but also between authors and reviewers on top of formal review reports and responses to those.

The tasks of reviewers can be structured to several extents, ranging from very open assignments with very little guidance to strict forms with obligatory assets to be scored using likert scales. Structuring the review process might be specifically relevant in case you want reviewers to check for compliance with reporting guidelines. Several studies have shown this compliance to be low, hence suggesting that more scrutiny is required (Baker, Lidster et al. 2014, Bandrowski, Brush et al. 2016).

In contrast, it is suggested that asking reviewers to score specific aspects of a manuscript potentially distracts their attention from other aspects, thereby effectively narrowing down the scope of review. Hence, structuring review can be particularly beneficial in case a journal wants to focus its review process on certain aspects of submitted manuscripts.

Statistics review can be performed in several ways, for instance by specialist statistics reviewers or by automated computer software. Both have been indicated to have certain benefits. In one of our studies we found review through specialist reviewers to be associated with fewer retractions (Horbach and Halffman 2019). Another meta-analysis found specialist statistics review to be favourable over ordinary review (Bruce, Chauvin et al. 2016). In addition, statistics review through automated software has shown its ability to detect reporting mistakes and inconsistency in reporting of statistical results (Nuijten, Hartgerink et al. 2016). Several such tools are currently under development or have been released over the past years.

In new initiatives, review is not necessarily performed by the journal to which the article is submitted. These initiatives most notably include the formation of journal consortia in which reviews of manuscripts rejected at one journal are transferred to other journals. This allows to reduce the number of times a manuscript needs to be reviewed (Horbach and Halffman 2018). In addition, through cooperation between journals, manuscripts might be directed towards the most relevant outlet. However, concerns have been expressed that such cascading review practices might enhance the stratification of publication outlets, favouring traditional journals and potentially hindering the introduction of new outlets.

Besides cooperation with other journals, it also possible to use reviews from commercial platforms and reviews performed by the wider community. The use of reviews from other (partner) journals is related to fewer retractions (Horbach and Halffman 2019). A pilot shows the ability of commercial review platform to detect integrity issues (Pattinson and Prater 2017).

There are several technical applications that could be of help in review, such as, plagiarism scanners, software to detect image manipulation, and software to check references (Burley, Moylan et al. 2017, Scheman and Bennett 2017, Horbach and Halffman 2019). Even though hard-evidence of their benefits is usually scarce, the use of technical support is related to fewer retractions (Horbach and Halffman 2019). In addition, the use of digital tools instead of a manual check could also save time. Examples of digital tools for peer review are: plagiarism detection software, digital tools to assess validity or consistency of statistics, digital tools to detect image manipulation, digital tools to check references, and upcoming machine learning techniques to assess consistency and completeness.

The facilitation of reader commentary seems to be a useful mechanism to flag issues, and hence contributes to the quality of the published record (Horbach and Halffman 2019). Reader commentary could either be facilitated in-channel, for instance through comment sections on your journals’ webpage. Effectively, this constitutes a form of post-publication review. Alternatively, commentary could be facilitated out-of-channel using other platforms where readers can provide feedback or comments to published manuscripts. Several of such platforms have emerged over the past decade and increasingly provide an additional regulating mechanism in science (Biagioli and Lippman 2020).