Here is his abstract:

There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

You’re quite right that the calculations I did ignored—purposely—the prior probabilities of the studies being true. I wanted to highlight the non-intuitive, and even surprising, result that so many false studies could be published given that all effects are actually zero. But you’re right to stress that this is not the whole answer to the question.

That the p-value is a conditional probability is nearly always under-appreciated. To emphasize (for our readers): it is the probability of seeing a larger statistic than what you got given the effect is zero. But it is impossible, even forbidden, in classical statistics to calculate what you actually want to know: (1) the probability the effect is non-zero given the actual data observed, and what you rightly point out (2) the probability the result is false given that it is published. P-values simply cannot answer these questions. What is shocking, however, is that you are not even allowed to ask them in the first place! (In classical, frequentist statistics.)

Now, you mention a particular article in which “The p-value is well below 1 in 10,000″ which is “highly significant.” There is trouble here, because all we know is that we are unlikely to see a particular value of some statistic given the effect is 0. Unfortunately, a low p-value does not always imply a high probability that the effect is non-zero.

Regarding your last point, i.e., that about “a 1% chance that any given published study has incorrectly statistically significant results”, there has been some work on this in the published literature; I think, the British Medical Journal and also the on-line PLOS: Medicine. I’ll have to dig up these articles to quote more exactly, but one of these guessed that as many as 50% of all studies are incorrect! I think that estimate is high, but only because I am an optimist.

Though I’ll soon make a posting about a standard analysis practice that is bound to give a false result, so perhaps I should be more of a pessimist.

Ill get to the second point below, but the as for the first: when you limit the analysis to published studies, all the p-values are less than .05. In studies published in major journals, I would imagine most of them are much lower than 0.05, thus indicating a conditional probability of a lot less than 1 in 20.

Let’s take the lead article of the current issue of the NE journal of Medicine. This article (see http://content.nejm.org/cgi/content/short/358/1/9 ) talks about the effect of delayed time to defibrillation. The total sample size is more than 6,000, of whom more than 2,000 had delayed defib. The p-value is well below 1 in 10,000. NEJM studies are certainly more likely to be highly significant, so I am not suggesting it is indicative, but we do tend to have more faith (yes I saw your post on this word) in journals like NEJM, and one of the reasons why is that they tend not to publish borderline results.

My next point has to do with which probability to measure. The p-value is the probability of getting a result at least as different from the “no effect” result as the one obtained, given that the true state of the world is that there is no effect (I can work to get the language better, but we both know the definition so I’ll move on). What we are really looking for is the probability that there is no effect, given the study is published. I tend to think that in better reviewed journals, reviewers will be wary to publish startling new results without some outside proof or even independent replication. Even ignoring this, the proabbility relies in part on what percentage of research studies are of something that truly has no effect. If we assume this is one-half, then around 1 in 20 studies published have spurious results, supporting your claim (consider 1000 studies, 500 with no effect and 500 with an effect; assuming a “power” of 95%, about 475 of the “with effect” studies will get published and at an alpha of 5% about 25 of the no effect studies will get published). But in order to do these studies, you have to get funding, etc, which is difficult to do. If this extra layer of difficulty means that, in fact, 80% of research is on something that has a non-zero effect, then we have closer to a 1% chance that any given published study has incorrectly statistically significant results.
-Alan Salzberg

AGW =Anthroprogenic Global Warming

Look forward to the post referred to above.

Methinks that one should read all reports with a sceptial eye, not just medical.

Cheers

That error rate is a direct consequence of using the p-value approach. It is because it is true that there is a 1 in 20 chance of seeing a statistic as large or larger (in absolute value) than the one you happened to get, given that no effect exists.

You cannot guard against it, either, in the peer review process. After all, both the people who wrote the paper and the reviewer see a p-value less than the magic number. So both sides assume that the effect is real, and the data you have do seem to support it (using classical statistics).

Nowhere in classical statistics can you answer the question: “What is the probability that the effect is real?” That always surprises people, but I’m afraid it’s true.

I’ll soon post an article about how you can make even bigger mistakes if you follow a very standard procedure, one that is almost guaranteed to give you a p-value less than the magic number, but where all the data is completely made up.

(What’s “AGW”?)

Briggs

I am surprised that there is even the likelihood that medical reports ie peer reviewed papers, would have such a high error rate even at the rate of 1:20, or about 5000 in total.

Doesnt say much for the peer review process,when lives are at risk.Imagine what it would be for AGW science

Cheers

Malcolm Hill
Adelaide