Twitter @ceptional reminded me of this post, which I had forgotten. Since it is highly relevant to The Great Bayesian Switch, I decided to repost. Some minor errors in grammar have been corrected.
The Twitter user @alpheccar asked me to look at the discussion with the same name at Stack Exchange. We’ve covered this before—a statistical pun: see below—but it’s worth another look.
It helps to have an example, so here is a trivial one. I want to quantify my uncertainty in the heights of male U.S. citizens, from age 18 until dead. This will let me answer such scintillating questions as, “What is the probability that a man (meeting this demographic condition) is taller than the perfect height of 6’2″?”
I decide, based on reasons unfathomable to the lay mind (but really because the software allows me to), to quantify my uncertainty in heights using a normal distribution. I do not say “Heights are normally distributed”, because they are not. Heights are determined by various biological and environmental causal factors. But I can say, “My uncertainty in heights is quantified by a normal.”
Normal distributions are characterized by two parameters, a central and a spread, usually labeled with the Greek letters μ and σ. In order to characterize my uncertainty in height, I must needs supply values for these parameters.
I can, if I like, just guess what these parameters are. Why not? I can set μ = 5’8″ and σ = 3″, and who are you to say I am wrong? Can you prove I am wrong? You cannot. In this case, via my argument by authority, there is no question of confidence or credible intervals. My guesses are fixed and final.
But I am humble and decide to be amenable to empirical evidence, so I grab a sample of men (alive now) and measure their heights. Because I am a mathematical wizard, I can compute the mean and standard deviation of this sample. There is no uncertainty in these calculations, or in this sample. If I want to know the probability of a man in this sample being taller than the average, I just count. There is still no question of credible or confidence intervals.
What have the mean and standard deviation to do with the values μ and σ? Not a damn thing. Unless I embrace frequentist theory, which allows me to substitute these empirical measures as guesses of the parameters. And recall I need guesses, otherwise I cannot quantify my uncertainty in heights.
No frequentist believes that these guesses of the parameters are perfect, without error. Here is where confidence intervals arise: through a formula, I can compute the 95% confidence interval for these guesses. But first, an old joke:
“Excuse me, professor. Why a 95% confidence interval and not a 94% or 96% interval?” asked the student.
“Shut up,” he explained.
What is a confidence interval
The confidence interval formula supplies a numerical interval for both the guess of μ and σ. But if I wanted, I could go back and regather a new sample, and I could compute a new guess of μ and a new guess of σ, and new numerical intervals. Agreed?
Well, I could do it a third time, too, producing a new set of guesses and intervals. Then a fourth time, then a fifth, and so on ad infinitum. Literally. Still with me?
Here is what a confidence interval is: 95% of those intervals from the repeated samples will “cover”, or contain, the true values of μ and σ. That’s it, and nothing more.
But what about the interval I calculated with the only sample I do have? What does that interval mean? Nothing. Absolutely, positively nothing. The only thing you are allowed to say is to speak the tautology, “Either the true values of μ and σ lie in the calculated intervals or they do not.”
What is a credible interval
The Bayesian way is not satisfied with a mere guess of μ and σ. Instead, prior information about the values of these parameters is gathered or assumed and used to probabilistically quantify the uncertainty in their values before any data is gathered. That is, before new data comes in, I can ask questions like, “Given this prior information, what is the probability that μ < 5″?”
But after I gather a sample, and through the magic of Bayes’s rule, I can calculate the uncertainty I have in the values of μ and σ accounting for this data. I can then picture the complete distribution of uncertainty of both parameters, or I can pick an interval of values where I think the true values of the parameters are most likely to lie.
Here is what a credible interval is: Given the data and the model, there is a 95% chance the true values of μ and σ lie in that interval. That’s it again, and nothing more.
But what about the heights of those men?
We forgot! Just like everybody else who does a statistical analysis, we got distracted by computing guesses and intervals that we plumb forgot what our original purpose was. How silly of us! We were so happy that we knew the difference between confidence and credible intervals that we abandoned our original question and spoke as though the certainty we had in the parameters translated to the certainty we had in heights. But of course, this is miles from the truth. See this post.