Well, just a little (all emphasis mine and joyfully placed):
Statistical significance is junk science, and its big piles of nonsense are spoiling the research of more than particle physicists…
But here is something you can believe, and will want to: Statistical significance stinks…
The null hypothesis test procedure is not the only test of significance but it is the most commonly used and abused of all the tests. From the get go, the test of statistical significance asks the wrong question…
In framing the quantitative question the way they do, the significance-testing scientists have unknowingly reversed the fundamental equation of statistics. Believe it or not, they have transposed their hypothesis and data, forcing them to grossly distort the magnitudes of probable events…
They have fallen for a mistaken logic called in statistics the “fallacy of the transposed conditional.”
And that’s just the first part. I couldn’t finish the second because my eyes were overflowing with happy tears.
Ziliak and pal Deirdre McCloskey, incidentally, co-authored the must-read The Cult of Statistical Significance.
Cult, they say. Cult because there is an initiation at high price. Cult because statistical “significance” is invoked by occult incantations, the meaning of which has been lost in the mists of time. Cult because these things can not be questioned!
The p-value is a mysterious, magical threshold, an entity which lives, breathes, and gazes sternly over spreadsheets; a number gifted to us by the great, mysterious god Stochastikos1. It was he who decreed that great saying, “Oh-point-oh-five and thrive; Oh-point-oh-six and nix.”
Adepts know the meaning of this shorthand. So 0.050000001 is sufficient to cast a result outside the gates where there is weeping and gnashing of teeth. Yet 0.04999999 produces bliss of the kind had when the IRS decides not to audit.
Members cannot be identified by dress but by their manner of speaking. Clues are evasiveness and glib over-confidence. They say, “The probability my hypothesis is true is Amen” when what they mean is “Given my hypothesis is false, here is the value of an obscure function—one of many I could have picked—applied to the data assuming the model which quantifies its uncertainty is certainly true and that one of its parameters is set to zero and assuming I could regather my data in the same manner but randomly different ad infinitum.”
In the hands of a master, more significant p-values can be squeezed out of a set of data than donations Al Sharpton can secure by marching into an all-white corporation’s board room.
“Statistically significant” does not imply true nor useful nor even interesting. “Significance” is a fog which emanates from a computerized thurible, thick and pungent. It obscures and conceals. It woos and insinuates. It distracts. It is a mathematical sleight-of-hand, a trick. It takes the eye from the direct evidence at hand and refocuses it on the pyrotechnics of p-values. So delighted is the audience at seeing wee p-values that all memory of the point of a study vanishes.
Statistical significance is so powerful that it can prove both a hypothesis and its contrary simultaneously. One day it pronounces broccoli as the awful cause of splentic fever and tomorrow it is asserts unequivocally that broccoli is the only sane cure for the disease.
Both results will be accepted and believed, especially by those manning (and womanning!) bureaucracies and press rooms. Journalists won’t tell you about the deadly effect of either until 10 p.m. Government minions will latch gratefully on to anything “significant” as proof their budget (and therefore power) should be increased.
Time for statistical significance to be slain, its bones cremated, and its ashes scattered in secret. No trace should remain lest the infection re-spread. The only word of it should appear in Latin in tomes guarded by monks charged with collecting man’s (and woman’s!) intellectual follies.
Thanks to Steve E for finding Ziliak’s piece.
1I didn’t think of this; I recall the name from the old usenet days.