Here is how peer review roughly works.
An author sends a paper to a journal. An editor nearly always sends the paper on to two or more referees. The referees read the paper with varying degrees of closeness, and then send a written recommendation to the editor saying “publish” or “do not publish.” The editor can either accept or ignore the referees’ recommendations.
The paper is then either published, or sent back to the author for revisions or rejection.
If the paper is rejected, the author will usually submit it to another journal, where the peer review process begins anew. This cycle continues until either the paper is published somewhere (the most typical outcome) or the author tires and quits.
Here are two false statements:
(A) All peer-reviewed, published papers are correct in their findings.
(B) All papers that have been rejected1 by peer review are incorrect in their findings.
These statements are also false if you add “in/by the most prestigious journals” to them. (A) and (B) are false in every field, too, including, of course, climatology.
A climatology activist might argue, “Given what I know about science, this peer-reviewed paper contains correct findings.” This is not a valid argument because (A) is true: the climatology paper might have findings which are false.
If the activist instead argued, “Given what I know, this peer-reviewed paper probably contains correct findings” he will have come to a rational, inductive conclusion.
But a working climatologist (gastroenterologist, chemist, etc., etc.) will most likely argue, “Given my experience, this peer-reviewed paper has a non-zero chance to contain correct findings.” Which is nothing more than a restatement of (A).
The “non-zero chance” will be modified to suit his knowledge of the journal and the authors of the paper. For some papers, the chance of correct findings will be judged high, but for most papers, the chance of correct findings will be judged middling, and for a few it will be judged low as a worm’s belly.
Here is a sampling of evidence for that claim.
(1) Rothwell and Martyn (abstract and paper) examined referees’ reports from a prominent neuroscience journal and found that referee agreement was about 50%. That is, there is no consensus in neurology.
(2) No formal study (that I am aware of) has done the same for climatology, but personal experience suggests it is similar there. That is, there is at least one published paper on which the referees do not agree (at what is considered the best journal, Journal of Climate).
(3) Pharmacologist David Horrobin has written a commentary on peer-review in which he argues that the process has actually slowed down research in some fields. He also agrees with my summary:
Peer review is central to the organization of modern science. The peer-review process for submitted manuscripts is a crucial determinant of what sees the light of day in a particular journal. Fortunately, it is less effective in blocking publication completely; there are so many journals that most even modestly competent studies will be published provided that the authors are determined enough. The publication might not be in a prestigious journal, but at least it will get into print.
(4) I have just received an email “Invitation to a Symposium on Peer Reviewing” which, in part, reads:
Only 8% members of the Scientific Research Society agreed that “peer review works well as it is”. (Chubin and Hackett, 1990; p.192).
“A recent U.S. Supreme Court decision and an analysis of the peer review system substantiate complaints about this fundamental aspect of scientific research.” (Horrobin, 2001)
Horrobin concludes that peer review “is a non-validated charade whose processes generate results little better than does chance.” (Horrobin, 2001). This has been statistically proven and reported by an increasing number of journal editors.
But, “Peer Review is one of the sacred pillars of the scientific edifice” (Goodstein, 2000), it is a necessary condition in quality assurance for Scientific/Engineering publications, and “Peer Review is central to the organization of modern science…why not apply scientific [and engineering] methods to the peer review process” (Horrobin, 2001).
This is the purpose of the International Symposium on Peer Reviewing: ISPR (http://www.ICTconfer.org/ispr) being organized in the context of The 3rd International Conference on Knowledge Generation, Communication and Management: KGCM 2009 (http://www.ICTconfer.org/kgcm), which will be held on July 10-13, 2009, in Orlando, Florida, USA.
Be sure to visit the first link for more information.
(5) Then there is the Sokal Hoax, where a physicist sent a paper full of gibberish to a preeminent social science journal to see if it would be published. It was. Sokal was careful to play to the preconceptions of the journals’ editors to gain acceptance. The lesson is the oldest: people—even scientists!—easily believe what they want to.
(5) John Ioannidis and colleagues in their article “Why Current Publication Practices May Distort Science.” The authors liken acceptance of papers in journals to winning bids in auctions: sometimes the winner pays too much and the results aren’t worth as much as everybody thinks. A review of the article here.
(7) UPDATE. Then there is arxiv.org, the repository of non-peer-reviewed “preprints” (papers not yet printed in a journal). Arxiv is an acknowledgment by physicists, and lately mathematicians and even climatologists, that it’s better to take your findings directly to your audience and bypass the slow and error-prone refereeing process.
(8) It is easy to get a paper into print when the subject is “hot”, or when you are friends with the editor or he owes you a favor, or your findings shame the editor’s enemies, or through a mistake, or by laziness of the referees, or in a journal with a reputation for sloppiness. In most fields, there are at least 100 monthly/quarterly journals. Thus it is exceedingly rare for a paper not to find a home, no matter how appalling or ridiculous its conclusions.
Update: 4 March; Reader Jack Mosevich reminds us to see this article at Climate Audit. Real-life example of politically correct refereeing.
The listing of these facts is solely to prove that (A) and (B) are false, and that peer review is a crude sifter of truth.
Thus, when an activist or inactivist points to a peer-reviewed paper and says, “See!”, he should not be surprised when his audience is unpersuaded. He should never argue that some finding must be true because it came from a peer-reviewed paper.
This web page has also tracked several peer-reviewed, published papers that are crap. Examples here, here, here, and here (more are coming).
————————————
1Incidentally, I have only had one methods paper rejected; all others I wrote were accepted to the first journal I sent them to. Nearly every collaborative paper I co-wrote has also been accepted eventually. I am an Associate editor at Monthly Weather Review, and have been a referee more times than I can remember, both for journals and grants.
I mention these things to show that I am familiar with the process and that I am not a disgruntled author seeking to impugn a system that has treated him unfairly. To the contrary, I have been lucky, and have had a better experience than most.
Citing peer review is an appeal to authority.
Speed,
It is, but of the inductive, rational kind of appeal. It is only false to say “Because this article is peer reviewed, it contains a true finding.” It is perfectly rational to say, “Because this article is peer reviewed, it probably contains a true finding.”
But in climate science, it would be more accurate to say, “Just because this article is peer reviewed is not sufficient to conclude that the findings are false.”
With all due respect Stan…
And in climate science, it would be just as accurate to say, “Just because this article WAS NOT peer reviewed is not sufficient to conclude that the findings are false.”
Your invitation in (4) is just a piece of spam.
Briggs:
Personal observation: While such qualifiers are rarely used the first is often implied and the second seldom inferred.
For lent, I am not giving up anything, but taking up the habit of using the words “likely†and “probablyâ€. Let me start by making the following two statements.
* Mr. Sixteen Ounces probably should not argue that Mr. One Pound is of little weight.
* If an article is peer reviewed, it’s more likely to be logically correct .
Sven,
Well I’ll be dogged. I had no idea that “resume stuffing” conferences even existed (and me a cynic). Thanks much for pointing it out.
The references the email listed are real, however, and important to this topic, so I’ll leave up the email as is.
The continual references to “peer review” literature and the denigration of articles that are not “peer reviewed” seems to me to unique to the global warming debate. One also hears references to the “scientific consensus” much more frequently in the global warming debate then in any other areas of discourse. Did all of this arise from Al Gore emphasizing the now discredited review of literature that made the claim the there was no peer review literature that challenged the scientific consensus on global warming.
Also, may I refer readers to this interesting article on the limits of peer review by Bruce McCullough and Ross McKitrick http://www.fraserinstitute.org/commerce.web/product_files/CaseforDueDiligence_Cda.pdf
But what can be said for papers that are initially rejected, (B) above, and the try-another-journal schema gets it into (A)? It is then a peer-reviewed paper the finding of which is both correct ad incorrect.
Don’t both (A) and (B) implicitly assume the Certification status of the peer-reviewers; all peer-reviewers are qualified to peer-review papers and do not ever make mistakes?
At the bottom of the Wiki entry about Skal are lists of other hoaxes as well as peer reviewed research which proved false.
http://en.wikipedia.org/wiki/Sokal_affair
Peer review has its good and its bad.
But, the argument that I can’t suffer is when an activist argue that peer review is better than blog, when blogs, like climate audit goes through a stricter reviewing than journals.
The good thing about peer review or even just editorial review is that it raises the quality of articles so they are easier to read, thus more efficient for communication. This even helps the authors think better themselves. The problem is that scientists are lazy.
Where are the results of the election survey?
TCO,
You’re right about that. At least, it’s mostly the copy editing that’s done by the journal’s publisher. Though I have had reviewers who fancy themselves copy editors. They point out every misspelling, misplaced comma, and so forth. Pain the butt.
Thanks for reminding my about wishcasting. Been busy. Will do soon.
I’ve always thought peer review was a sort of poor man’s replication, arrived at sort of like this:
Start with “So-and-so has replicated my work and knows it’s right”
through “So-and-so has examined my work and thinks it’s right”
to “So-and-so has read my report on my work and thinks it might be right”.
I get the feeling that the confirmation conferred by replication is supposed to hang on somehow through this process of dilution.
Just a thought.
I never trust peer review until at least 30 years have passed. Sheep have a tendency to bleat when their neighbors bleat.
It’s worth pointing out (and it is implied, but not stated explicitly in Jack Mosevich’s comment) that the Sokal hoax was perpetrated on a journal which did not have peer review, at least not at the time. Granted, the paper was accepted by at least one editor, but this is not an indictment of peer review, per se.
Noahpoah,
Good point. But I believe there were actually several editors involved with Sokal’s paper (from my memory of his book), so it’s pretty close to the standard peer-review model.
Sorry to leave this here but I can find no other place to send a message regarding a technical error which is preventing either Firefox or Akregator both under linux from receiving the RSS blog feed.
Here is the error reported:
XML Parsing Error: XML or text declaration not at start of entity
Location: https://www.wmbriggs.com/blog/feed/
Line Number 2, Column 1:
^
It has been failing for me since last Friday.
cheers… jack
Jack,
Working in it. Supposedly, it’s a blank line in one of the php files. I got the comments rss back on, but I can’t find where the content bug is.
arXiv is the Way! You look for new cutting edge research by scanning arXiv for names you recognize. You look in journals for the current publishing trends.
I think the statement, “Peer review is central to the organization of modern science,” is true but somewhat misleading. It is true that grant output and the career progress of academic scientists are measured by counting peer-reviewed articles but that doesn’t mean that it is central to the modern practice of science (i.e.,comparing ideas to reality) or the organization of scientific ideas – which is how I see peer review sometimes presented.
Peer review as an institution is more a sociological phenomena among academic scientists than it is a process connected to the correct practice of science. Counting peer review articles is a way of removing individual judgment and that is the true goal of every government/university bureaucrat that was ever pupated.
“The listing of these facts is solely to prove that (A) and (B) are true, and that peer review is a crude sifter of truth. ”
Given that you introduced (A) and (B) as falsehoods surely you meant
“The listing of these facts is solely to prove that (A) and (B) are false”
—
Incidentally I don’t think the “even scientists!” fits with the Sokal hoax. I’m sure we could start a debate on whether social science actually is a science but I won’t do that. I will observe that the fact they fell for the hoax could be taken as evidence of their non-scientific status rather than evidence that scientists fall for nonsense.
TDK,
Whoops! Typo. Thanks. I’ll go up and fix it.
Plenty of real “scientists” have fallen for hoaxes before, too. Think of Uri Geller in the late 70s.
Yes no dispute with the fact that scientists have fallen for nonsense.
Think of the widespread support for eugenics in the 1920s amongst the scientific community. Or how quickly they bought Piltdown man.
However I think you tread on dangerous ground with Uri Geller type examples. I’d conjecture that he discovered a remunerative career as a showman and switched. He got a lot of media exposure but there was always healthy scepticism within science. You can see a similar phenomena with James Lovelock. He was a good scientist but the reception to the Gaia hypothesis showed great discomfort. He’s moved from science to popularism.
In other words Geller clearly ceased to be a scientist, Lovelock also but perhaps less overtly.
Social Scientists believe in the Post Modern doctrine of relative truth. There are plenty of practitioners who hold that things like the scientific method are uniquely western concepts hiding a power structure that privileges western ideas at the expense of those of other cultures. Thus Alan Sokal told them exactly what they wanted to hear. Moreover as an actual scientist he gave their ideas a credibility they could only aspire to.
I see peer review as a process to determine whether the findings in the study contain information worthy of being viewed by the public or by people in the field. I have suggested accepting papers with conclusions I did not fully agree with, but the data were sound, supported the claims and could provide benefit to the field and future studies. I have also reccommended rejection of papers making claims I believe to be true, but had poor experimental design and the data did not support the claims.
I have often wondered whether or not a truely original discovery would get published in a peer reviewed journal. That goes for tenure too; a really brilliant innovator might have difficulty getting tenure if the judges (faculty) do not appreciate or understand his/her work. It would also be very difficult to get funding if current othodoxy is challanged.
If the judges do not understand a truely brilliant piece of research it may never see the light of day. Are we perhaps perpetuating the status quo by having too much government funding and influence?
briggs…you should take a STRAIN. Clear writing is clear thinking. Clear thinking is HARD. Don’t be an air force slacker about it. Be an anal Navy nuke. Give it your absolute best effort. It is easier to find flaws or debate key issues…when the presentation is CLEAR.
There is a lot of literature out there. Be KIND to your reader (reviewer, editor, or eventual reader) and give him the best product you can. When you submit, what to you is PERFECT, then it is easier for people to improve it. And you will also be amazed how stuff flows through peer review when it is brutally simple and honest and clear in discourse (to include admitting dropped samples even) and completely rigorous in grammar/reference/typography as per the Notice to Authors.
Were the peer-review studies peer-reviewed before being published?