From our very own JMJ, who asked this in response to an announcement of my new book (read this first):
Briggs, how many people how you encountered who need these clarifications? I mean, just how many people misunderstand this stuff? You taught this stuff, so I guess you’ve seen how a lot of people see this. Is this a real intellectual problem out there among math students? Or are you seeing something that’s not there?
For instance, when most people say that something happened by chance, do you think they’re trying to say that they believe that a thing called chance actually made it happen? When someone says they see a trend in the numbers, do you think they’re trying to impart a belief that somehow the numbers themselves are creating a trend?
Excellent questions, all. Everybody who uses a hypothesis test, whether by p-values or Bayes factors, needs these clarifications, misunderstands the purpose of standard tests, and these misunderstandings are a real intellectual problem. They’re not a math problems—everybody’s math is accurate—but philosophical problems.
And some people, not all, really do believe that chance is causative. More people, not all, really do believe numbers, i.e. models, are creating trends. The remaining people who do not really believe, but use the old methods, fail to recognize that the methods they use logically imply chance is a cause and that the numbers/models are a cause. So in that sense, everybody gets it wrong.
Speaking only of observable models, what we want is this: Pr(Y | X, old data, assumptions). This is the probability of some observable Y given premises X (this may be multifaceted), old observations (which we might not have), and other assumptions (this usually includes includes ad hoc statements like “I will use a normal”).
I say: to communicate models use Pr(Y | X, old data, assumptions)!
Not too exciting an answer, right? I don’t think it is, either. But, except for a trivial minority, it isn’t what anybody does.
Instead, people will calculate a p-value, which takes the assumptions, which include ad hoc statements about unobservable parameters to models, and functions of the old data, and then calculates the probability these functions would exceed some value if the “experiment” which produced the old data would be infinitely repeated and assuming that parameters are equal to some value.
If the p-value is less than the magic number, people think that one of the Xs they picked was the cause of the data. Maybe not the cause of all the data, but of a “significant” portion of it. If the p-value is greater than the magic number, they say X was not a cause but “chance” or “randomness” was.
Some might not think they are saying these things, but they are in fact saying them by implication. To prove that is not difficult, but it takes more than 800 words, so I’ll leave it for the book. I have a hint in these two papers “The Crisis Of Evidence: Why Probability And Statistics Cannot Discover Cause“, and “The Third Way Of Probability & Statistics: Beyond Testing and Estimation To Importance, Relevance, and Skill“.
It’s not better using Bayes factors, because these are also statements about (functions of) the parameters, though with only some of them at fixed values. The same fallacy about cause is there. The same forgetfulness about Pr(Y | X, old data, assumptions)—which is all anybody ever wants to know, but which the classical methods ignore (except in rare cases)—is also there. This is why there are no “good” uses of p-values or Bayes factors. If you want the probability, go to the probability and skip the substitutes!
The Deadly Sin of Reification includes any kind of smoothing, or estimation of parameters, where the probability model is taken for reality. Since probability is not a cause, and neither it chance or randomness, the plotting of probability models over real data often (always?) leads people to think “what really happened” was the model and not the data. What really happened, or what’s really going on, are the data. If we knew the cause or causes of the data, then we don’t need the probability models. Why would we?
Understanding cause is our fundamental goal in science. With probability, we don’t need cause but can do predictions, which is the fundamental goal in engineering. So I don’t advocate dumping probability; I do say which should use it properly.