Jaynes’s book (first part): https://bayes.wustl.edu/etj/prob/book.pdf
Permanent class page: https://www.wmbriggs.com/class/
Uncertainty & Probability Theory: The Logic of Science
Link to all Classes. Jaynes’s book (first part):
Video
Links:
Bitchute (often a day or so behind, for whatever reason)
HOMEWORK:
Lecture
This is an excerpt from Chapter 5 of Uncertainty. All the references have been removed.
Probability Is Not Subjective
Cournot was right “le calcul des chances” is “un calcul des illusions.” Illusions are what allow the false belief that there is a unique probability number for any problem. And this would be all right, except that this has been allowed to develop into a separate theory of probability; i.e. the curious theory of subjective probability. This states that the right probability value, for any problem, is that one that “floats your boat” or that gives you warm feelings. It is not surprising that this theory developed in a century when relativism of every kind was in vogue.
Truth and probability are not subjective, all of whom makes the errors detailed here. The certainty of any proposition is to be adjudged only by the accepted evidence, just like any math problem must be solved only by the conditions set by an examiner. If an instructor said, “Given $x + y = 7$ and $y = -3$” and asks “Solve for $x$” and a student says, “I feel, oh so strongly, that the probability $x = 32$ is 100%” the student would, at least in the old days, be marked wrong. But a subjective probabilist would have to say to the student, “You must be right: your feelings are what count.”
When evidence which is not accepted or commonly or tacitly understood is injected into lists of premises, the certainty of the conclusion changes—as it must. This injection is why probability can appear subjective. Yet if probability were subjective then any conclusion would follow from any set of premises. Take “There are 10 Schmenges and 4 Minyks in a room and only one of these persons will come out”. We deduce “The probability a Schmenge comes out” as 5/7. But if probability were subjective we could say this probability is 0.01%” or any other number, including 0% or 100%, that gives us a “positive” feeling. Anchoring probability on feelings is a dubious idea: it admits indigestion could play a role in a proposition’s chances.
Since probability is not subjective, if the probability of a proposition is different than the one which is deduced (or deducible) from its premises, this always implies injected or substituted premises (or a simple mistake in calculation, which is the same thing).
If our evidence is “A heck of a lot of people like this product and Mary is a person”, the probability of “Mary likes this product” is not a number because the notion “A heck of a lot of people” is vague and unquantifiable. Yet some might still form a notion of the likelihood of Mary’s pleasure based on tacit premises regarding the meaning of “A heck of a lot…” These differing premises are what make it appear probability is subjective.
To prove that, let’s pick a homelier example. People gamble on sports. One person says the Tigers will win, another says they will lose. That a person gambles does not imply the person has fixed a probability for an event (as we’ll see below). But let’s suppose each of our two persons do form coherent probabilities for tonight’s game. Person A says the probability of victory is 80% and B says 30%. This makes probability appear subjective. We ask each how they came to their judgment. A recounts past games, and his comprehensive (he claims) knowledge of tonight’s pitchers. B admits the past games, but lays more emphasis on the batting. A discussion like this can range far and wide and last forever. Each party can admit the other’s points, but each might stick to, or adjust part way, his assessment. Again, probability seems subjective. But that is only because the premises are so many in number that it is difficult or impossible (and probably impossible) to show how the probabilities that are held are deduced from these premises.
And we can’t quite tease out whether the stated probabilities are actual deduced probabilities or decisions. They may well be decisions and not probabilities because not all premises imply single-number probabilities, or even numerical probabilities. In those cases, when a probability is stated, it is because a decision has been made or because additional premises have been added or others subtracted.
The key is that if two people agree exactly, precisely, and completely on a set of premises, and on how those premise are probative to the proposition of interest, they must, or rather should, agree on the probability. See also Aum1976, who proves the similar specialized proposition “If two people have the same priors, and their posteriors for an event A are common knowledge, then these posteriors are equal.” The result is much more general than considering priors and posteriors. Probability is only subjective in the weak sense that the choice of premises is not fixed. People are free, in many cases, and especially day to day, in choosing premises. But this freedom is, and should be, greatly reduced in science. The goal of science is to find just those premises which make a proposition as near to certain as possible, so there is no subjectivity.
Another simple problem, as above. Solve for $x$—give a single, unique number—in the following equation: $x + y = 3$. Of course, it cannot be done: under no rules of mathematics can a unique $x$ be discovered; there are one too many unknowns. Nevertheless, someone holding to the subjective interpretation of probability could tell us, say, “I feel $x = 7$.” Or he might say, “The following is my distribution for the possible values of $x$.” He’ll draw a picture, a curve of probability showing higher and lower chances for each possible $x$, maybe peaking somewhere near 3 and tailing off for very large and small numbers. He might say his curve is equivalent to one from the standard toolkit, such as the normal. Absurd?
It shouldn’t sound absurd. The situation is perfectly delineated. The open premise is that $x = 3 – y$, with a tacit premise that $y$ must be something. The logical probability answer is that there is no probability: not enough information. (We don’t even know if $y$ should be a real number!) But why not, a subjectivist might say, take a “maximal ignorance” position, which implies, he assumes, that $y$ can be any number, with none being preferred over any other. This leads to something like a “uniform distribution” over the real line; that being so, $x$ is easily solved for, once for each value of $y$. Even if we allow the subjectivist free rein, this decision of uniformity is unfortunate because it leads to well known logical absurdities. There cannot be an equal probability for infinite alternatives because the sum of probabilities, no matter how small each of the infinite possibilities is, is always (in the limit) infinity; and indeed this particular uniform “distribution” is called “improper.” Giving the non-probability a label restores a level of comfort lost upon realizing the non-probability isn’t a probability, but it is a false comfort. Aiding the subjectivist is that the math using improper probabilities sometimes works out, and if the math works out, what’s to complain about?
To say we are “maximally ignorant” of $y$, or to say anything else about $y$ (or $x$), is to add information or invent evidence which is not provided. Adding information that is not present or is not plausibly tacit is to change the problem. If we are allowed to arbitrarily change any problem so that it is more to our liking we shall, naturally, be able to solve these problems more easily. But we are not solving the stated problems. We are answering questions nobody asked.
Subjective probabilists make several errors, sometimes singly and sometimes in concert. These are: to add to or change the premises or evidence, to confuse probabilities with decisions or acts, to assume the propositions which receive the probability must be physical “events,” and to assume all probabilities must be numerical.
The first and most blatant is, as was just said, to add to or change the given premisses. Gamblers, even intelligent ones who well know the rules of the games they are playing, are notorious for this, saying certain numbers are “due” or that others are “over played.” Even if they don’t act, they change the accepted premises to accommodate their superstitions about the probabilities. Bayesian statisticians often invent “priors” (which we’ll discuss later) to accompany ad hoc probability models, these being necessary to solve the equations. But these “objective” “maximum entropy”, “ignorance”, or “reference” priors are not (or almost always are not) suggested by the given premises.
Now these inventions are more or less harmful, and even at times useful if the premises guessed or invented turn out to match reality, in the sense that the premises which led to assigning probabilities where none was previously possible are agreed on and provide useful decisions. But it is always the case that adding premises changes the problem. Adding any information not tacitly plausible—such as we did when assuming more meant at least half and not all, or even when we assumed that all meant each and every one without exception—is to answer a different question. It is not to play the same game. I emphasize this because experience has shown that this point is difficult to accept. People see little trouble adding whatever information they desire to stated, fixed problems merely so that they can arrive at a solution. However practical this is, it is not answering the stated, fixed problem, but answering a new, self-created one. Subjective probability in this sense is like the student on a high school algebra example saying (in the last example), “I think $y=3$ therefore $x$ must be 0” and that student expecting to be rewarded for his perspicacity. The premise “$x+y = 3$” is not equivalent to “$x+y = 3$ and $y = 3$” or “$x+y = 3$ and $y \in [-4,4]$” or anything else.
Typical justifications for subjective probability, thanks mainly to Bruno De Finetti and Frank Ramsey, involve Dutch books. In the simplest example, there is an “event” of interest. This event is a proposition, usually of the form E = “X will happen at such and such a time and place.” We want the probability of E. There are no stated premises, so the problem is not solvable—unless we add premises; in this case, unstateable subjective feelings. Now if our man says that, given his feelings, the probability of E is 0.7, but that given these same feelings the probability of not-E (X will not happen) is 0.5, then our man’s emotions have led him astray; his probabilities are said to be incoherent because they do not sum to 1. Whatever method of elicitation we use must lead to coherence, or Dutch book can be “made” against him. The man faces sure loss.
His incoherence made plain to him, our man is now invited to think in terms of money or of its mental equivalent, “utility”, a fictional currency made of, it appears, discretized emotions. A price for a ticket is set such that our man will pay \$1 (or some other amount; however, this figure makes the math easy) if E obtains, and nothing if it doesn’t. His opponent, Nature, also given to gambling, gets to choose either side of the bet; that is, Nature can choose to either buy or to sell the ticket, and the man must accept Nature’s decision; his only freedom is in setting the price. Obviously, the price must be \$0 if the man thinks E impossible, and \$1 if he thinks E certain. If the man thinks E impossible and offers a price of (say) \$0.4, Nature will decide to sell the ticket. Since, to the man, E will never happen, he will be out \$0.4; thus if he truly thinks E impossible, the only reasonable price is \$0. The idea is that considering his feelings and knowing Nature gets to decide which side of the bet the man is on, he will set the price which best reflects his idea of E’s likelihood. The price (since the payoff is \$1, or otherwise suitably normalized) becomes the “probability.”
This works as a gamble because if the man’s stated probability is not coherent, in the sense noted above, Nature can make a sure profit against the man. For example, if the man’s stated probability for E is 0.7 and for not-E is 0.5, Nature could sell the \$0.7 ticket for E to happen and sell the \$0.5 ticket for E not to happen. If E occurs, Nature looses \$0.3 on the ticket for E to happen, but it gains \$0.5 on the ticket for E not to happen, for a net profit of \$0.2. If E does not occur, Nature gains the full \$0.7 on the ticket for E to happen, but looses \$0.5 on the ticket for E not to happen, again for a profit of \$0.2. If the man is incoherent, Nature must necessarily win. Incoherence is a Dutch book, or Nature’s arbitrage.
Now none of this tells us whether the man has set his probability for E soundly, even if coherently. Soundness is another matter entirely. But you can now see that setting a probability based on emotional evidence does not give a probability but instead is a decision, an act, a bet. Bets are not numerical probabilities, no more than words like might and likely are numerical probabilities. They are good indicators of propensities, markers of behaviors and proclivities, but they are not themselves probabilities.
These betting justifications for probability are backward: Dutch book theory works because of probability; they do not define probability. Instead of defining probability, what they show is that probability is coherent, and that which is not probability might not be. Coherence surely does not guarantee a profit! The empirical bias in probability theory also shows here. There is no betting on the state of the Metalunan interocitor mentioned last chapter, and no Nature to take bets, simply because there are no Metalunans. But there is still probabilities of interocitors taking states.
To amplify that last objection, let Q = “There are exactly 100 Martians and only one wears a hat and George is a Martian.” The probability of P = “George wears a hat” given Q is 0.01. But a subjectivist can say, “Based on my utility, it’s 83.7%!”, or whatever. How can you prove him wrong? There are no experiments that can be run because there are no Martians. There are thus no bets that can be made, because there is no “event” to occur or not. Unless probability is treated as logic, you have nothing to say to the subjectivist and must accept his probability as being right, which is absurd.
Subscribe or donate to support this site and its wholly independent host using credit card click here. Or use the paid subscription at Substack. Cash App: \$WilliamMBriggs. For Zelle, use my email: matt@wmbriggs.com, and please include yours so I know who to thank.
