Reader and colleague JH has rightly taken me to task (via email) for incorrectly calling, or rather misleadingly labeling, logical probability “objective Bayes.”
She pointed to this set of lecture slides (pdf) as examples of what most people think of when they hear “objective Bayes.” Its basic idea is to use (these are a technical terms) “non-informative”, “references, or “ignorance” “priors” on unobservable parameters to get the math to work out in Bayes’s theorem.
I accept this.
For somebody (me) who’s always carping about precision in technical language, I also admit I earned my spanking for any confusion I have caused by this mix-up.
There is overlap between objective Bayes and logical probability. Both use Bayes’s theorem. But then so do frequentists use it when the mood strikes. Still, I can see now that it is improper (there’s a joke in that word) to call logical probability “LP Bayes” as I have been doing.
Saying “Objective Bayes” puts one in mind of Jeffrey’s priors, sequences of priors, improper, proper, and conjugate priors; information matrices, Markov Chain Monte Carlo and “drawing” observations; invariant measures, Radon Nikodym derivatives, with probability 1s, lemmas, theorems, proofs; unobservable parameters; parameters, parameters, and more parameters, parameters galore; a deeply mathematical subject which sometimes is and sometimes isn’t interpretable.
Of course, some of this stuff is useful to LP. Some isn’t. None of it is wrong mathematically; but then nothing is wrong mathematically with frequentism, either. Logical probability is not a branch of mathematics, though math is useful to it. Objective Bayes, at least academically, is math twenty-four hours a day.
Not that there’s anything wrong with that! Why, some of my best friends are mathematicians. Please don’t sic Anthony Kennedy on me: I have not formed an “improper animus” toward our calculating friends. God undoubtedly made them that way, and who am I to judge?
But I do have plenty of animus toward the reification of the mathematics. Take for example “improper priors.” These are government-defined “probability distributions”, which means probability distributions which aren’t probability distributions but are called probability distributions by those anxious to get on with the math. If you don’t already know, I can tell you that they are needed when jumping the infinity shark. Frequentists are rightly suspicious of them and are the reason some frequentists have not yet come over from the Dark Side.
By a curious coincidence, which ought to make both OBs and Freqs (shorthand is easier, isn’t it?) nervous, use of improper “flat” priors often produces identical mathematical results in both theories. Normal-distribution linear regression is the most prominent example. The interpretations still differ of course—which proves that probability in any flavor is not a mathematical subject.
Advertised by its name, logical probability is a branch of epistemology just as classic logic itself is. The biggest difference between LP and all the others is admitting what everybody already knew to be true: that not all probability is quantifiable and that probability is always a measure of knowledge, even in it mathematical disguise (the “always” carries much weight).
There is an enormous literature in this field that lies undiscovered and unread by nearly all statisticians. The best and most well known work which bridges the gap between logical and mathematical probability is E.T. Jaynes’s Probability Theory: The Logic of Science. But this is not to say that Jaynes’s book is the best on logical probability.
Another classic is A Treatise on Probability by John Maynard Keynes (yes, that Keynes), which opens with this definition:
Part of our knowledge we obtain direct; and part by argument. The Theory of Probability is concerned with that part which we obtain by argument, and it treats of the different degrees in which the results so obtained are conclusive or inconclusive…
The terms certain and probable describe the various degrees of rational belief about a proposition which different amounts of knowledge authorise us to entertain. All propositions are true or false, but the knowledge we have of them depends on our circumstances; and while it is often convenient to speak of propositions as certain or probable, this expresses strictly a relationship in which they stand to a corpus of knowledge, actual or hypothetical, and not a characteristic of the propositions in themselves.
For years, I’ve been trying to get statisticians to read David Stove’s The Rationality of Induction (especially its second half), but so far I have convinced just one Named Person (a prominence who resides in a university and in the blogosphere) to scan it—and he immediately proceeded to misinterpret it (those few statisticians who think about it love the so-called problem of induction; like all pseudo philosophical problems, it’s a guarantor of papers written to be read only by the writers of these kinds of papers).
The shortest introduction in here, here, and here. When you read these posts, understand that I was using objective in the sense of deduced or true, even if you don’t want it to be and not its mathematical sense. I know I also have to clean these up and collect them under one heading. Coming soon!