I stole today’s title from David Papineau’s essay “Thomas Bayes and the crisis in science“, which many readers sent in.
When I was in grad school bad in the early to mid 1990s, Bayes was just off its flush of becoming respectable, which occurred mostly in the 1980s. But then, as now, and as you’ve all heard me lament before, all statisticians must first be initiated into frequentism. As such, they find it difficult to overcome. The experience is not unlike trying to leave the religion of your youth. Sure, you can stop practicing it. But you can never stop feeling its influence.
This is why you still hear from self-styled Bayesians admonitions to develop Bayesian procedures with “good frequentist properties”, which is (a) begging a Simpson’s paradox-type situation, and (b) incoherent. If Bayes is right (about which sense more in a moment), then it’s always right and frequetism wrong, and vice versa. The two are not compatible philosophies of probability.
See Uncertainty: The Soul of Modeling, Probability & Statistics for more on all this, incidentally.
Anyway, Bayes has three interpretations. The subjective which says, and I do not jest, probability is a function the indigestibility of your food. The probability of any proposition is how you feel about it. It is therefore an effeminate philosophy (do not confuse feminine with effeminate). The objective, which is frequentist in character, and which thinks probability is ontic. This is a mistake. And then the logical, which says probability is epistemic. This is the correct view (which is not really called “Bayesian” by anybody, though people use it that way). I’m not proving this here: I’m telling you. Read the book for arguments.
The importance of Bayes is not—as I have stressed hundreds of times, to little avail—is not in the formula. It is not strictly needed, not ever. It is nice, it is helpful. But that is it. What we always want is
Pr(Y | X)
where Y is the proposition of interest and X is the totality—I’d shout this if I thought it would do any good—of evidence. This probability is not always quantifiable. Tough cookies. How we get to Pr(Y | X) is only of interest to technicians, and is where the formula might be of use. But it is always beside the point.
Which means all the ya-ya-ya about “updating beliefs” is beside the point. First, subjective probability is wrong, and second, the update is a technical matter. What always counts is the totality of evidence you accept. And the evidence you accept is not necessarily the same as I accept—or the same as anybody else accepts. Hence disputes. Probability is only a dull function of the evidence accepted.
The real revolution in Bayesian thought is that everything uncertain can be assigned a probability, though not always in number. There is nothing wrong with that sentiment, and everything right. But like I just said in other words, it is the evidence which counts. And only the evidence. The math connecting evidence to probability (the least interesting aspect) we can leave to geeks and nerds.
This is why we know statements like the following (from the article) are false in the strict sense:
Bayes’s reasoning works best when we can assign clear initial probabilities to the hypotheses we are interested in, as when our knowledge of the minting machine gives us initial probabilities for fair and biased coins.
No. What works best is assembling the evidence that comes closest to showing the cause of the proposition of interest Y. The wrong wrong has already been chosen, as we see by the next sentence “But such well-defined ‘prior probabilitie’ are not always available.”
We don’t need “prior probabilities” on the theory that some thing causes heart attacks. We need evidence that it does or doesn’t. Sometimes we start out ignorant. So what? We build evidence from that ignorance.
Thinking Bayes is a panacea, or a universal formula, is why die-hard frequentists are still scared of leaving their incorrect theory of probability. No panacea exists. Subjectivism is silly. And they are right.
But it is a false dichotomy to insist on either subjective/objective Bayes of frequentism. There is a third way.