The rules of logic are simple. Use whatever evidence you have, and no other, to figure the probability of a conclusion. These rules are all we need to understand what is a model and what makes it good.
Given just that (our model) E1 = “All Martians wear hats and George is a Martian” what can we say about C = “George wears a hat”? We deduce that C is true, that it has probability 1. Change the evidence upon which we base our conclusion and we can change the probability. Given E2 = “No Martians wear hats and George is a Martian” then C is false (probability 0); or given E3 = “Half of all Martians wear hats and George is a Martian” then the probability of C is 0.5
Nowhere in this analysis is there word one about whether E1, E3, or E3 (our possible models) are themselves true. We cannot know whether they are true or false unless we supply evidence upon which we judge them. In the arguments just used, we accepted that each were true in turn.
Suppose we have not yet met George and we want to guess whether he will wear a hat. We have three sets of evidence; that is, we have three different models. Models are just the stuff that is given, the evidence we accept as relevant or probative to the conclusion.
Well, we have three models E1, E2, E3. We might have learned each E from three friends who have visited Mars, each swearing the veracity. We might have dreamed them. It is (now, anyway) irrelevant how we learned them, we only need to know that it is these three, and no other, models that we must consider. Which model is the true model, which two are false?
We have asked another probability question, which we already know how to answer. What is our evidence? M = “We have three models, only one of which is true.” Given this M(eta evidence), what is the probability that E1 is true? One-third, and also one-third for each of the other two. This quantification can actually be derived and is the result of what is called the statistical syllogism (see this paper). But I hope it is an obvious answer.
That’s is. We’re done, at least as far as judging the probability of each model. Our only evidence was M, and we have said all there is to say given M.
But what about old George? Before we meet him, we’d like to know the probability he wears a hat. Accepting in turn the truth of each model, we know that this probability is 1, 0 or 0.5. But since we’re not sure which model is true, we must account for this uncertainty. Skip the derivation if you like; the answer turns out to be a weighted sum of the individual probabilities. Thus
Pr(C | M) = Pr(C | E1 & M)Pr(E1 | M) + Pr(C | E2 & M)Pr(E2 | M) + Pr(C | E3 & M)Pr(E3 | M)
Pr(C | M) = 1 * 1/3 + 0 * 1/3+ 1/2 * (1/3) = 1/2.
And that’s the best we can say given only the evidence in M and each of the E. In learning about C, we are well and truly finished.
Now George comes along and he’s either wearing a hat or he isn’t (a tautology). If he is, then what can we say about the individual models? If he isn’t, what then?
If George is wearing a hat (Borsalino snap-brim fedora, fur felt, brown), adding this new evidence to M, we know that the probability of E2 is 0. We have falsified E2. Falsification is rare for models concerning the contingent (physical and counterfactual events like Martians wearing hats). In order to falsify a model the model needs to say an event is impossible—not unlikely, but impossible—and then have the event happen. Model E3, for instance, can only be falsified if we line up all Martians simultaneously and discover that any proportion of Martians not exactly equal to one half wears a hat.
Since George has manned up and is wearing a hat it could be that E1 or E3 was the true model; E2 was false. Each model is, of course, entirely useless to us now, since we have seen George and already know that he wears a hat. Why do we care which is the true model? Well, we don’t really. Maybe we want to hand out a reward to the model that did “the best” in predicting George’s hat status. E2 is out of the money, but E1 or E3 might take the prize.
Well, E1 predicted George would certainly wear a hat and he did. Pretty good performance! But E3 said it was essentially the flip of a coin, which isn’t too bad, though not as good as E1. If you had beforehand bet that George would wear a hat (that C would be true), you would have won using either E1 or E3, but probably won more using E3 since if any bookie actually believed E1 he wouldn’t accept any money except from people who bet that C would be false.
The criterion of goodness is then a subjective matter, depending on what use you put the probability. Different folks can come to different conclusions about model goodness based on different uses.
We can go further, however, and ask a different question. We could turn the probabilities around and ask what the probability a model is true (say E2) given C (George wears a hat) and M (there are three models, etc.). In notation (and using Bayes’s rule and the rule of total probability; these rules are deduced truths from simpler axioms):
Pr(E2 | C & M) = Pr(C | E2 & M)Pr(E2 | M) /
[Pr(C | E1 & M)Pr(E1 | M) + Pr(C | E2 & M)Pr(E2 | M) + Pr(C | E3 & M)Pr(E3 | M)]
This follows from Pr(C | M ) = Pr(C|E1 & M)Pr(E1|M) + Pr(C|E2 & M)Pr(E2|M) + Pr(C|E3 & M)Pr(E3|M).
Plugging numbers in let’s us deduce Pr(E2 | C & M) = 0. We also know that Pr(E1 | C & M) = 2/3, and Pr(E3 | C & M) = 1/3. Please note, and please assimilate to the depths of your soul, that these probabilities have nothing to do with future instances (“random trials”) of seeing George. C states that George wears a hat and that is that. Given that we have seen that C is true, C is true and will stay true (given the evidence we have). If C were false (if ~C were true) then we would have falsified E1; E3 would have been just as likely and E2 would have (given the evidence) probability 2/3.
That, dear reader, is finally final. When probability is (rightly) seen as logic, every problem is treated the same. There is no difference between the treatment of these simple models and more complex ones. In Part II, we’ll look at more complex ones.