Why are peer reviews private?

That is a question raised by the experience of a “rogue” neuroscientist profiled in Wired today:

SAM NASTASE WAS taking a break from his lab work to peruse Twitter when he saw a tweet about his manuscript. A PhD in cognitive neuroscience at Dartmouth College, Nastase had sent his research out for review at a journal, and hadn’t yet heard back from the scientists who would read the paper and—normally—provide anonymous comments. But here, in this tweet, was a link to a review of his paper.

“I was like, ‘Oh that’s my paper, OK.’ So that was a little bit nerve-wracking,” says Nastase. A few weeks later, he received the same review as part of a response from the journal, “copied and pasted, basically.”

So much for secret, anonymous peer review. The tweet linked to the blog of a neuroscientist named Niko Kriegeskorte, a cognitive neuroscientist at the Medical Research Council in the UK who, since December 2015, has performed all of his peer review openly. That means he publishes his reviews as he finishes them on his personal blog—sharing on Twitter and Facebook, too—before a paper is even accepted.

Having just written a book on scholarly publishing — Scholarly Publishing and its Discontents (available now on Amazon) — you might have thought that I had thought about his issue but I haven’t (perhaps the reason why reviews have been luckwarm). I figured I had best make up for that omission here.

Under the current system, reviews of journal articles are kept private. They are between the editor, the reviewers and the author. When you think about it, it is not clear whether this is a norm or a rule but let’s evaluate it as if it is a rule. There are other norms out there like “don’t send out tweets like the following:”

@AEAjournals Our paper on the value of twitter complaints was rejected by AER. Sad! https://t.co/KdVtwLN9fa

— Joshua Gans (@joshgans) February 28, 2017

but I digress.

The privacy rule is almost certainly justified in terms of “preventing embarrassment.” But whose embarrassment? Let’s consider the authors. If someone writes a critical review then the authors won’t be happy and may prefer the state of the world where the review is private. But this is normally not what concerns us with regard to efficiency. Might it not be argued that authors would take more care before sending papers off for costly peer review if they knew they might be subject to criticism? We don’t keep conference interactions private and they can be much the same. Moreover, how many times have you received a critical review where you might like it to be public and thus give over an avenue to provide counter arguments and a dialogue. This is certainly what appeared to happen with Kriegeskorte’s efforts.

Let’s turn to the reviewers. They actually wrote the review so, in a sense, they own it. Why might they not want the review public? For the same reason as the authors. They may not want embarrassment or similarly may not want to have to take the same level of care they might do if it is not published. Now in this situation, efficiency suggests publication again so to improve care but there are some tricky issues.

First, reviews are anonymous and potentially for good reason. And, in principle, there is no reason why an anonymous review cannot be published although one might be concerned that the same issues that give rise to anonymous Internet commentors might arise (not that they can’t arise internally either). Of course, a reviewer may agree to have their name associated with the review. If this were optional then that fact may present a signal to the outside world of the reviewer’s beliefs and confidence. As the anonymous remain anonymous, there would be no cost to them.

Second, if reviewers were pressured to publish their reviews, they may refuse to do them. This is a very legitimate concern. It is hard enough to get reviews anyway let alone if you potentially raise the ‘costs’ of doing so. Then again, if they are anonymous, what costs are those precisely? The license to speak your mind and do so without care is still there. However, there may be some negative utility if others evaluate the review and find faults with it — anonymous or not. This doesn’t seem to be a certain enough concern to prevent some experimentation.

The final party who may be embarrassed is, of course, the editor. Published reviews mean that, at the very least, the editor — who is not anonymous — is subject to scrutiny. This may make it hard to find editors willing to do the job but remember those editors are acting in public service anyway. When you have a person who is in a position of power, we normally want to shine as much light as possible on them. At least that is how we err in such trade-offs. But for academic journals, we have gone fully for non-transparency.

One potential issue is that the editor may be looking for an independent read of the paper. Now what that means in this day and age when there are pre-prints, conferences, seminars and the like is hard to say. But if a paper had been rejected by a journal and submitted to another, the editor in the next round may have a hard time getting an independent evaluation. This suggests, at the very least, that perhaps the author might have some sort of right to prevent the review’s publication but I think that argument seems practically quite weak.

I have to say that I am hard pressed to find reasons why at least publishing your own, written reviews if you want to is a bad thing. Moreover, if the reviews are available, when a paper is rejected, those reviews will be available to other journals as well. This could really help reduce costs and delays in the system.