I flatter myself that this is the best general thing I have written to explain why you must never, not ever, use a p-value, hypothesis test, “significance”, Bayes value, or any of that.
It comes in review of the book Bernoulli’s Fallacy: Statistical Illogic and the Crisis of Modern Science by Aubrey Clayton.
Bernoulli’s fallacy happens when you give the probability of the data and assume it is the probability of the hypothesis. (Or functions of the same.) That’s what p-values, hypothesis testing, and so forth are. Mistakes. Always mistakes.
Let me quote myself, and point you to the Academic Questions site. I know clicking an extra button is a pain in the keister, but I am begging you will do so. JUST CLICK HERE.
After some time passes, on some quiet day I’ll republish the whole here so that I myself don’t lose track of it. I added the bold below to catch your eye. Because I really want you to read this.
Ask a psychologist what the chances are that a person will walk slower after reading from a list of words having to do with old age than reading from a neutral list. He won’t tell you. He can’t tell you. What he can tell you is that in his model of walking time, after “controlling” for a number of items including the word list, the “parameter” representing something to do with walking time was highly “statistically significant,” with something called a “p value” that was boastfully small.
You see a drug commercial on TV and are impressed by the cavorting of the actors. You want to cavort. So you go to the doctor and ask him if Profitol is right for you. You ask him the chance the pill will let you cavort. He won’t tell you. He can’t tell you. What he can tell you is that he read about an experiment using the pill, and that if the “null hypothesis” comparing that pill to another pill was true, the probability of seeing data that was not seen in the experiment was pretty low.
This answer being incomprehensible, you seek a second opinion. The next doctor gives you a test for cavortitis, the malady which causes an inability to cavort. The test is positive. So you ask the doctor, “Does that mean I got it?” He says, “Well, in those patients with cavortitis, the test comes back positive ninety-five percent of the time. And it’s even better for those without the disease: the test comes back negative ninety-nine percent of the time.” He writes you an exorbitantly expensive prescription for Profitol. Suddenly you don’t feel so good.
And you shouldn’t. Because the second doctor didn’t answer your question. Neither did the first. Neither did the psychologist. Neither can anybody who uses classical statistical procedures. Because those are designed not to answer questions put to them in plain language.
Take the second doctor. You asked him (implicitly) what the probability is that you have the disease after testing positive. Let’s call your having the disease your “hypothesis.” He instead tells you the probabilities having to do with the test. The test is “data,” so he gives you probabilities of the data instead of the probability of the hypothesis. Worse, he acted as if the probability of the data were the probability of the hypothesis. So did the first doctor and the [psychologist].
So does everybody who uses classical statistical procedures.
The conflating of the probability of the data as if it were the probability of the hypothesis is called, as Aubrey Clayton tells us in the book of the same name, Bernoulli’s Fallacy.
Once again, I am asking as a big favor to JUST CLICK HERE.
We—you and I, together, dear reader—would improve Science inestimably if we could only convince others of this lesson. I beg your help. Please pass on this page or the Academic Questions page to any who quote any science paper that uses “significance” etc.
Oh, why the “[psychologist]” in brackets? Because my original example used a climatologist. But then I decided that might trigger some weak minded critics, so I changed it to a real-life psychologist example. And then my enemies, wee P worshipers all, put back “climate scientist” for “psychologist”.
Subscribe or donate to support this site and its wholly independent host using credit card click here. Or use the paid subscription at Substack. Cash App: \$WilliamMBriggs. For Zelle, use my email: matt@wmbriggs.com, and please include yours so I know who to thank. BUY ME A COFFEE.
