I and a few like-minded folks have written many times of the over-certainty which is all but guaranteed using classical statistical methods. By “classical” I mean the ubiquitous frequentist p-value-centric “hypothesis testing” framework. But I also mean parameter estimation-focused frequentist and Bayesian methods.
Both testing and estimation take far too much for granted. Every analysis begins by assuming more than is warranted, a predicament explained by the impulsive rush to quantify that which is unquantifiable because it is felt that only quantification is scientific, and the analysis ends with a result in which too much credence is granted and too much faith is placed.
I won’t here rehearse the multitude of arguments and examples against classical approaches, nor will I outline the superior alternate approach (which many call “predictive statistics”). I will only state that this method is designed to better state the level of certainty one should have in a problem, and that this certainty is always less than traditional scheme.
One would have wagered that since the predictivist philosophy is superior that it would have been adopted. It has not because it is hard to “sell” somebody on the idea of less certainty. “Sure, you can use the classical stuff, which implies you should be 95% sure of your result. Or you could use the predictive method which says your belief should be just higher than a coin flip, and maybe even less than that.”
Who the hell wants to buy a product which claims it will deliver less? The urge to “be sure”, to have a decision made for you by “objective” criteria is too strong. Besides, “everybody else” is using the other stuff. Why shouldn’t you?
Whereas classicists promise clear skies, predictivists forecast fog. Classicists offer resolution, predictivists blurred vision. The classicist wants to get on with it, the predictivists says hold on a minute. The illusion of certainty often trumps the promise
It’s not like predictive methods haven’t made inroads. Casinos us them and always have. Automated data processors like license plate readers, handwriting recognition, and barcode scanners are so routine they don’t even seem like statistics, but they are1. These triumph because they are simple. Saying whether a scribble is an “f” or a “h” is trivial next to explaining (say) why a woman has an abortion. Even voice recognition—a notoriously “difficult” problem—is tinker toys when compared to saying how much the economy will expand or contract next quarter, let alone in a decade.
Yet there is no shortage of economists (folks somebody once called “statisticians without personalities”) willing to tell you exactly what the GDP will be on October 10th, 2022. Just are a plethora of “soft” scientists convinced that their theory of ___ism (where the blank may be filled in with the political concern of the day) is verified by a regression model—which the press will call “a computer model.”
Some say “soft” scientists—educationists, sociologists, psychologists, and so on—are envious of the prestige of mathematicians and physicists, the two professions (in order) which can rightly boast of confidence in their results. The certainty “quants” have, like we talked about before, is because these professions picked easy subjects.
Saying why a proposition is true because certain others are, once you’ve identified the new proposition, is a matter of mental elbow grease. And explaining why a certain particle moves in a field where all the variables are precisely known and controlled takes almost no brain power. Not compared to saying what a person—even worse, what people—will do and why he or they do it six months from now.
I don’t think it is envy, but habit which drives the “soft” scientist (or other typical statistics user) to his over-confidence. Everybody does the same thing he is doing, and from that he develops his confidence. “It can’t be wrong if so many people are winning so many grants and publishing so many papers.” It’s not easy to change a custom, especially a beloved one.
1Yes, it’s all statistics, that is, all probability, even though it sometimes goes by other labels and done by (say) computer scientists. See this post for an explanation. Or see many posts on the Classic Posts page.