It was the Nines last night. Pizza, beer, and wine, followed by sketchy music.
A definite conceit of statistics is its habit of declaring with something-approaching-certainty that this or that hypothesis is true. It relies much too much on quick and easy answers, or on mathematical cleverness. Nearly always lost are demonstrations that what was predicted actually occurred.
Ever read an academic paper in one of the areas that rely almost exclusively on statistics? Fields like sociology, epidemiology, and the like. They all “run” some statistics, present some p-values, make some definite conclusions.
But how often do you see a follow-up article which says something like, “We tested the model we built in our last paper on data we had never seen before (in any way), and here is how it fared”? I’ll tell you how often: never.
These kinds of papers are common in more concrete fields, of course, areas in which prediction of real things are fundamental.
It’s too easy to find the result you’re looking for with statistics. This is echoed in a review of the book Wrong: Why Experts Keep Failing Us—And How to Know When Not to Trust Them today at the Wall Street Journal.
But the current market creates the wrong kinds of incentives for doing good research or admitting failure. Novel ideas and findings are rewarded with grants and publication, which lead to academic prestige and career advancement. Researchers have a vested interest in overstating their findings because certainty is more likely than equivocation to achieve all of the above. Thus the probability increases of producing findings that are false. As the medical mathematician John Ioannidis tells Mr. Freedman: “The facts suggest that for many, if not the majority of fields, the majority of published studies are likely to be wrong.”
Look up Ioannidis’s name and read some of his papers. You’ll be glad of the time spent reading them.
That’s it for my warning. Class dismissed.
Regular service resumes tomorrow.