Philosophy

Peer Review and Proof

Suppose that I today, below, in this post, prove to you a certain mathematics theorem, say, “The Symmetry of Individual Constants.”

Is that theorem true?

Before you answer, consider that you probably aren’t a professional mathematicians or logician. Since you are not, you might not understand each step in the proof. Or if you are a mathematician, because you are inexperienced in the particular field, you might not understand the shorthand or the “givens” routinely used.

Also, and most importantly, the proof has not been “peer reviewed.” Which is to say, it has not been vetted by others who do understand the history and shorthand of this particular field.

Given all that, I repeat the question: is that theorem true?

Of course it is. This was not a trick question. I said—my main premise was—that I proved the theorem. A proof means that the truth of a theorem has been demonstrated. Quot erat demonstradum, and so forth.

Specifically, it does not matter one whit whether you believe the result, nor does it matter whether scores of people believe it. Even stronger, it does not matter whether experts in the field know of, approve of, or have reviewed the result.

We operate under a simple tautology: either my proof is valid or it is not. Whether that proof has been vetted is irrelevant. The theorem is true or not. Moreover, it has always been true or not. It was even true before I discovered a proof of its validity. Human agency does not—and cannot—change the truth status of the theorem.

Obviously, then, where a valid proof appeared is irrelevant to the proof’s validity. Another way to say this is that it is a logical fallacy to claim, “Because this result was not peer reviewed and published in an academic journal, it is false.”

This fallacy is routinely used by academics who cannot refute a distasteful result. “If So-and-So is so confident in his work, let him publish it!”

Our friend Steve McIntyre hears this fallacy a lot. McIntyre has published on his blog many true statements about the climate that are disliked or are unwanted by mainstream climate scientists. Instead of trying to answer, or to refute, McIntyre’s results, these global warming advocates dismiss them because they have not been peer reviewed and published.

Unfortunately, this fallacy—as shockingly obvious as it is—is usually accepted as proof of refutation. That is, it is thought that a refutation has been demonstrated merely because a credentialed scientist cast an aspersion.

Journalists—God love ’em—are particularly eager to swallow this faulty line of reasoning. I have yet to see a case where a reporter rebuts, “Yes, McIntyre’s work hasn’t been published. But how is it wrong?” That is, of course, the only question worth asking.

This fallacy does not just find a home in climate theory: it is warmly accepted everywhere. It’s familiarity is proof that scientists have egos just as large as Hollywood stars.

Have you heard of Grigory Perelman, the bushy eyebrow-bearing Russian mathematician who has been turning down prizes?

In 2002, Perelman wrote up a valid proof of the Poincaré conjecture, probably the most lusted after unsolved problem in mathematics. Briefly, imagine a two-dimensional surface on which is placed a loop of string. If you can shrink that loop continuously until it reaches a point such that the loop nowhere becomes kinked or stuck on a hole on the surface, then that 2D surface is really the same as a sphere; albeit a sphere that might be pulled, crushed, and twisted out of shape.

The Poincaré conjecture says that this same loop-shrinking is true for three-dimensional spaces. And what’s the most familiar 3D space? Well, Space. The universe. Not for the first time, Poincaré had said something deep about life, the universe, and everything.

Point is, the non-conformist Perelman submitted his results to the non-peer reviewed, non-journal Arxiv.org. This is a place where scientists (you have to be recommended to win publishing privileges) place their thoughts in advance of, or in lieu of, peer review. It’s a place to publish not-so-polished, non-popular thoughts in the hope of winning First to Discover status or to avoid the tediousness of peer review.

It turns out that a group of Chinese mathematicians wanted in on Perelman’s success. They re-wrote Perelman’s results and published them in a peer-reviewed, academically approved journal. They also strongly hinted (in a 2003 Science article) that theirs was the real proof—so that credit properly belonged to them—because their proof had been formally published.

The incident is non-technically retold in the New Yorker, where it now seems that Chinese group have recognized their use of the not-published fallacy. They are now claiming “they couldn’t follow” Perelman’s shorthand, even though other mathematicians could. Well, academics was always a blood-stained enterprise.

Anyway, let’s get to work. How many instances of the not-published fallacy can we find?

Categories: Philosophy, Statistics

16 replies »

  1. If unpublished means false, doesn’t that imply that anyone who hasn’t been published is not, in fact, a scientist, and therefore is unfit to be published? This would, taken to its logical conclusion, lead to the death of science within a generation…

  2. gcb:
    I am confused. I thought Matt’s question referred to whether a non-peer reviewed written finding should be given priority over a finding submitted for publication at a later date in a peer reviewed journal.

  3. Newton and Leibnitz?
    But then we get into a host of other issues around completeness, independence, etc. Your example of a proof is less ambiguous than a discovery or an invention or a method.

  4. gcb,

    There is an out. If you have never been published, you must have a previously published co-author to establish your legitimacy.

    There is a value to peer review. If I don’t have the skills to review the work myself, what other tools do I have to build my confidence in the conclusions? While it would be wrong to conclude that that which has not been reviewed is necessarily suspicious. I should be able to say that, if it has been peer reviewed, I should have higher confidence that the conclusions are correct. Unfortunately, far too many times have the outlandish and the incorrect cleared peer review and been published.

    My biggest gripe with the publication process is that non-results don’t get published. If 10 people say that X has no influence on Y, and 1 person finds the opposite, who gets the ink? This gets compounded when someone runs a meta-analysis.

  5. I once “derived” a model that violated the second law of thermodynamics. It is publish and no one in the intervening 35 has ever noticed the error. I will never reveal where to find it, but I am thoroughly ashamed and learned my lesson. Ten years ago I fought valiantly to move a paper through peer review, and one of the reviewers blocking publication kept insisting that I alter an equation into a form that would violate the second law. I never managed to get this paper published.

    What can we say about peer review? It is probabilistic.

  6. Are you not in danger of approaching, if not crossing, the line of conflating the proof of a mathematical theorem with a non-maths theory, which cannot be proved? The requirements and status of the two are different. As you say, if each step of a maths proof is demonstrably true, then the final result holds (as long as nobody mentions Goedel). But that doesn’t work outside of mathematics, where are the axioms?

  7. The phrase is “Quod erat demonstrandum”, meaning “That which had to be shown”. “Quod” is the neuter singular nominative/accusative of the relative pronoun “qui, quae, quod”, meaning who/which/non-interrogative what.

    “Quot” on the other hand is an indeclinable Latin interrogative which means “How many?”

    “Demonstrandum” is the neuter singular nominative/accusative form of the gerundive of the verb “demonstrare”, to show. The gerundive, a verbal adjective, is used in Latin to express need, “should/needs to be/must/have to” as in Cato the Elder’s [in]famous injunction which he appended to all his speeches on any and all topics: “Carthago delenda est” – Carthage must be/should be destroyed.

    Old Cato would have given Dick Cheney a run for his money any day, NTTAWWT.

  8. A related point: “peer review” has nothing whatever specifically to do with science. It is simply a convenience for journal editors; you are likely to find a peer review process similar to Nature’s at a journal devoted to, say, 16th century Italian poetry.

    The distinguishing characteristic of scientific publishing has always been complete openness and explicitness about data and methods; that’s why so many 18th and 19th century scientific papers were accompanied by elaborate drawings of the experimental apparatus. And this is still the norm, everywhere but “climate science” — which I find a powerful indication that whatever Mann, Jones & Co. are doing, it isn’t science…

  9. Hi Matt,
    I’m sure you saw Dr Walt Meier’s response to Willis’ post, answering the 14 (16) questions.

    I would be interested in your take:
    Your own answers,
    Your thoughts on Dr. Meiers answers, particularly addressing the logic of and behind his Null Hypothesis 2 (NH2).

    Thanks
    Hal

  10. Hal,

    No, I hadn’t seen this. I did not even know of the existence of the questions.

    They are pretty easy, though, and I’ll and answer them. Maybe Sunday or Monday.

  11. Matt,

    When you ask, “Is the theorem true?” (given that you have legitimately proved (proven?) it) the answer is not unequivocally “yes”. It depends on the truth values of the axioms used in your mathematics, or, if you are talking about mathematical theorems that relate to the real world, it depends on whether your axioms represent reality. “The shortest distance between two points is a straight line” seemed true to Euclid, but in Reimann’s universe, which is really ours, that shortest distance is the segment of an arc.

    Very much looking forward to your response to Willis’s questions.

  12. MrCPhysics,

    Quite right. We can never prove the truth of an axiom; they are made known to us through our intuitions. And sometimes things that appear axiomatic, are later proven not to be, as you example demonstrates.

Leave a Reply

Your email address will not be published. Required fields are marked *