In a recent study, a greater fraction of Whites than Blacks were found to have a trait thought desirable (or undesirable, or a trait thought worth tracking). Something caused this disparity to occur. It cannot be that nothing caused it to occur. “Chance” or “randomness” are not operative agents and thus cannot cause anything to occur. It might be that we cannot know what caused it to occur, or that we guess incorrectly about what caused it to occur. But, I repeat, something caused this difference.
If you like, substitute “Pill A” and “Pill B”, or “Study 1” and “Study 2”, etc. for White and Black.
I observed a greater fraction of Whites than Blacks possessing some trait. Given this observation, what is the probability that a greater fraction of Whites than Blacks in my study possessed this trait? It is 1, or 100%. If you do not believe this, you might be a frequentist.
What is the probability that the proportion of trait-possessing Whites is twice—or thrice, or whatever—as high as Blacks in my study? It is either 1 or 0, depending on whether the proportion of trait-possessing Whites is twice (or whatever) as high as Blacks. All I have to do is look. No models are needed, no bizarre concepts of “statistical significance.” All we need do is count. We are done: any empirical question we have about the difference (or similarities) of Whites and Blacks in our study has probability 1 or 0. It is as simple as that.
Now suppose that we will see a certain number of Whites we have not seen before; likewise Blacks (they could even be the same Whites and Blacks if we believed the thing or things that caused the trait was non-constant). We have not yet measured this new group of Whites and Blacks so that we do not know whether a greater proportion of Whites than Blacks will be found to possess the trait. Intuition suggests that since we have already observed a group in which a greater proportion of Whites than Blacks possessed the trait, the new group will display the same disparity.
We can quantify this intuition with a model. There are many—many—to choose from. The choice of which one to use is ours. All the results derived from it assume that the model we have chosen is true.
One model simply says, “In any group of Whites and Blacks, a greater proportion of Whites than Blacks will be found to possess the trait.” Conditional on this model—that is, assuming this model is true—the probability there will be a greater proportion of trait-possessing Whites than Blacks in our new group is 1, or 100%. This simple model only makes a statement about Whites possessing the trait in higher frequency than Blacks. Thus, we cannot say what is the probability the proportion of trait-possessing Whites is twice (or whatever) as high as Blacks in my study.
Some models do not let you answer all possible questions.
We could create a model which dictates the probability that we find each multiple (from some set) of fractions of Whites than Blacks (e.g. twice, thrice, 1/2, 1/3, etc.), and then use this model to make probability statements about our new group. Since that would be difficult (and somewhat capricious), we could instead parameterize the differences in proportion.
We could use this model to answer the question, “Given this model is true, and given the observations we have made thus far, what is the probability that the parameters take a certain value?” This question is not terribly interesting and it does not answer what we really want to know, which is about the differences between Whites and Blacks in our new group. Why ask about some unobservable parameter? (The right answer is not, “Because everybody else does.”)
But given a fixed value of the parameters, we could answer the question, “Given this parameterized model is true, and given a fixed value of its parameters, and given the observations we have made thus far, What is the probability a greater fraction of Whites than Blacks will posses the trait?” This is almost what we want to know, but not quite, because it fixes the values of the unobservable parameters.
Simple mathematics allows us to answer this question for each possible value of the parameters, and then weighting the answers by the probability that the parameters take those values (this is from the parameter posterior distribution, which is conditional on the model being true and on the observations we have made thus far). The final number is the probability that the fraction of Whites is larger than Blacks in our new group. Which is what we wanted to know. (This is called the predictive posterior distribution.)
“Statistical significance” never once enters into this or any real decision. When you hear this term, it is always a dodge. It is an answer to a question nobody asks and nobody wants to know. It always assumes, as we do, on the truth of a model (though it remains silent about this, hoping by this silence to convince that no other models are possible). It tells us the probabilities of events that did not happen, and asks us to make decisions based on probabilities of these never-happened events. If you want to be mischievous, ask a frequentist why this makes sense. Homework: Locate Jeffreys’s relevant quote.
See the first in this series to discover what to do if we suspect our model is not true.