Peer review vs commercials and spam

Posted on 16 June 2010 by Stephan Lewandowsky

Guest post by Stephan Lewandowsky

Suppose a guy drifts up to you in a cafe and says that, like Julius Caesar, he’s a universal genius and that he’s willing to give you some brilliant investment advice. Would you entrust him with your retirement savings?

Probably not.

Now suppose your dentist tells you that you need a tooth capped soon in order to avoid a much more painful and extensive root canal job in a few years. Would you get the tooth capped?

Probably.

We all know that there are differences in credibility between opinions, with some carrying much more weight than others. Most of the time, we have an intuitive understanding of whom to trust and whom to dismiss. But is there a better way to find out whether a piece of information should be trusted?

When it comes to scientific issues of any kind, including climate change, there is no better source of information than the famous “peer-reviewed” literature.

Why is that? And what exactly is peer review?

The concept of peer review is about 300 years old and is the cornerstone of modern science. The best analogy for peer review is that it acts as a spam filter: rubbish ideas are kept from being published so that other scientists don’t waste their time reading spam. Only ideas that are not obviously rubbish make it into the literature, and once in the literature, the scientific marketplace of ideas determines their ultimate fate.

Let’s examine how different this process is from just posting one’s ideas on some web page.

Suppose I am a scientist and I believe that I have made a remarkable discovery; for example, I may believe that I have developed a new vaccine against swine-flu in my laboratory. I then write a paper that reports those results and submit it for publication to a peer-reviewed journal.

Writing a scientific paper is no trivial task: For example, in most cases, I must report my method in sufficient detail for other scientists to be able to replicate my experiment. I cannot keep my method secret; I must explain precisely what I did and how and with what equipment and for how long and so on. I typically must state who is funding my research and I must declare conflicts of interest, if any—for example, if I own the company that produces the equipment for my experiment, then I must typically disclose that. I also have to specify exactly how I analysed my data so that some other scientist could, in principle, repeat not only my experiment but also repeat the data analysis.

Now I submit my paper and a long waiting period begins.

The editor of the journal sends the paper to several other scientists. Their job is to independently, and usually anonymously, evaluate my work and to determine whether my experiment was sound and is worthy of being published.

This is the heart of peer review: Other scientists examine whether something is good enough to survive scrutiny and to be published.

Oh, and don’t be fooled by the name “peer” review; those scientists might be my peers, but they are definitely also my competitors! They have every intention of critiquing my work because they have no reason to give me a competitive edge—so there is nothing chummy about peer review; on the contrary, it’s as competitive as the Olympics.

After a few weeks, the reviews have been completed, and the editor now makes a decision on their basis. I was an editor for three years during which I made 300 editorial decisions. Sometimes a decision would take days because so much hinges on it: If an editor rejects a paper, someone’s career may suffer a set-back. If an editor accepts a rubbish paper, then the reputation of the journal is compromised; so it is important to get it right.

The best journals in the field pride themselves in the high proportion of papers they reject: It is not unusual for a top journal to reject 90% or more of all submissions, and the very best journals in the world, Science and Nature, reject even more than that.

For a paper to be rejected, all that is required is for one of the reviewers to find a flaw: Science is not a democracy, so if 2 reviewers like a paper and only 1 reviewer recommends against publication, that’s irrelevant—if the 1 critical reviewer has found a fatal flaw, then the paper is history.

Science is based on merit, not on democracy.

This sounds tough, and it is.

But it is that ruthless focus on merit and quality that has enabled science to deliver the accomplishments we all benefit from: It was peer-reviewed science that developed the anti-retroviral drugs that can now control HIV. It was peer-reviewed science that discovered the physics that got us to the moon, and it was peer-reviewed science that identified the potential threat posed by climate change and that also delivered the knowledge necessary to deal with the problem.

Of course, like any other human endeavour, peer review is not without its problems: It is possible that some papers that were rejected later on turn out to be noteworthy after all. Or, far more likely, bad papers may make it into press that should have been culled. Luckily, those bad papers don’t last very long because other scientists publish rebuttals or commentaries that reveal the flaws—thus putting egg on a lot of faces, including those of the reviewers and editors.

Here are a few more things about peer review that most people don’t know or tend to forget: First of all, there is no commercial consideration involved in the publication decision—none, zero, zippo, zilch. This is because the people who make editorial decisions stand nothing to gain or lose from a publication decision. And the journals are published either by professional organizations that don’t need to make a profit, or if they are private publishers, they make their profit by selling the journal to university libraries. And those libraries will subscribe to journals regardless of whether or not Dr. Smith or Professor Jones gets to publish his favourite theory.

In other words, making or losing money does not enter into a peer-reviewed publication decision.

Compare that to the decision about whether or not a book will get published: Harry Potter was published because the publisher thought they could make money. If they had thought that the book was a loser, they wouldn’t have published it—as indeed many of them did before J K Rowling found a publisher. So even when a scientist publishes a book, it is a commercial venture from the publisher’s point of view, and the quality of the author’s science remains to be ascertained, for example by reviews of the book by other scholars.

Compare peer reviewed science to the tabloids, which will print virtually anything to drive up their sales! If a quick lurid lie can sell another few thousand extra copies, sure thing, it’ll be in the headlines. If misrepresenting climate science is currently fashionable, sure enough, the tabloids will go out of their way to turn things upside down to make a buck, never mind what the science actually says.

And finally, compare the process of peer review to the people who run websites that sprout nonsense about conspiracy theories ranging from “MI6 killed Princess Diana” to “9/11 was an inside job” and “Climate change is a hoax.” Well, the only hoax is played on those people gullible enough to put any credence in that nonsense.

So now you know.

Peer review is a spam filter. Peer review is quality control. And peer review is independent of commercial interests. No wonder peer review gets us closer to the truth, even if that truth is inconvenient.

NOTE: this post is also being "climatecast" by Stephan Lewandowsky on RTR -FM 92.1 at 11.30 AM WAST today. It should air a few minutes after this post goes live so if you're reading this immediately (eg - you've subscribed to the SkS mailing list and just got this email), you can listen online via http://www.rtrfm.com.au/listen.