Since the subject arose yesterday, and for other reasons which I’ll explain later, I thought we should revisit this series, which ran a year ago. I have edited and expanded the text (just this Part so far) taking into account user comments. Plus I realize that I never provided a promised 4th installment, so I’ll do that too.
A certain English lady claimed to be able to tell whether her tea or milk was poured first into her cup. The statistician and geneticist Ronald Fisher put her to the test by presenting her four cups with the tea poured first and four cups with the milk poured first. The lady did not know the order of the cups.
Can we use this experiment to discover whether the lady has the ability she claims? We only have the evidence of this one test. The situation seems straightforward enough, but it isn’t. The difficulty lies in defining has the ability (see the Sorites Paradox Isn’t for what can happen when definitions aren’t paid sufficient attention). We cannot afford to be sloppy here.
Which of these best describes “has the ability”:
- She always guesses correctly (she is never wrong),
- In any experiment with N cups, she always gets at least N/2 right,
- In any experiment with N cups, she might get at least N/2 right,
- She always guesses correctly when the tea is poured first, but will sometimes guess wrongly when the milk is poured first,
- She always guesses correctly when the milk is poured first, but will sometimes guess wrongly when the tea is poured first,
- She guesses all cups correctly until the Mth cup (M < N), after which her palate becomes fatigued. M may depend upon a host of factors, such as the time of day, the food she at earlier that morning, her mental attitude, and so forth,
- She guesses at least M/2 cups correctly until the Mth cup (M < N), after which her palate becomes numb, etc.?
We could have expanded this list easily. For example, “She always guesses at least W/2 cups correctly when the tea is poured first, but will sometimes guess wrongly when the milk is poured first, where she is presented with 2W = N total cups.” Some of these lead to tricky counting, because if, say, she always guesses the tea-first cups correctly, and these come first in the sequence, and she assumes she knows these guesses are correct, after she sees N/2 cups she knows all the rest will be milk-cup first and she will therefore guess accordingly.
None of these definitions is in any way strange: each could really be what we mean when we say this lady knows her elevenses. “Hey!”, you might ask, “Where is the classical ‘She guesses better than chance?'” Are you sure it’s not already there? Let’s see.
The phrase guesses better than chance must be an idiom, because chance is not causative; that is, chance cannot be presented with cups of tea and asked to guess. So what is it idiomatic for?
Imagine an experiment where you are presented with N cups, but you do not touch, sniff, taste, or see inside these cup. You do not even see or know who places them in front of you; indeed, the cups can be left in a distant room, miles away from you. However, you must still make a guess whether the tea or milk was poured first into these occult cups. You could guess none right, or just 1, or just 2, and so on up to all N. What is the probability that you guess none right? Because our evidence (or premises) do not specify any known causal path for you to guess correctly, and because there is a natural ordering of guesses, we deduce the probability you guess any individual cup correctly equals 1/2. As long as you are not told whether your prior guesses were correct, this probability remains fixed.1
In particular, you are not asked to guess the sequence, but whether tea or milk was poured first; i.e. we want to know the number of your correct guesses and are not interested in the order of these guesses. Also notice that there is no information in these premises that suppose there will be an equal number of tea-first and milk-first cups. But even if there were, even if we knew there were equal numbers of each and thus that there were 2N possible sequences of cups, we are still not interested in the probability of your guessing sequence. For those who know, the number-sequence distinction is what allows us to pick between Carnap’s c* and c+ measures (a stumbling block for some, which at one time caused skepticism over logical probability).
The number you guess correctly—given no causal path—thus follows a binomial (if we don’t know how many of each cup; if we do, see Part IV). Importantly, you could guess, and we could figure the probability of your guessing, none right, or just 1, or 2, or even all. So, “guessing by chance” must mean the ability to guess any number correctly. Since you can and will guess some number (even all) correctly, you cannot “guess better than chance.” There is circularity. No matter if you get 0 right, 1 right, up to N right, all are consistent with guessing by chance. But we have at least learned that “by chance” means “by no (known) causal path.”
Now suppose it’s you against the lady; same lack of causal path for you, and her using all her powers. Who will win? If she always guesses correctly, then at best you could only match her. The probability of matching is (1/2)N, which makes the probability of her beating you 1 – (1/2)N. We deduce this assuming she never fails. Similarly, if we assume that “had the ability” means that “in any experiment with N cups, she always gets at least N/2 right”, and although the math is slightly more complicated, we could also calculate the probability of you tying, losing to her, or even winning.
We could go through each of our definitions of “has the ability” (and more like them) and calculate probabilities of you winning, losing, or tying. But none of these exercises tells us which of these definition is true, or which is more likely true than another. For that, we must turn our thinking around.
Read Part II.
1This becomes extremely important in, say, ESP experiments. See Persi Diaconis (who first taught me of this) and Ron Graham’s, “The analysis of sequential experiments with feedback to subjects” in the Annals of Statistics, 1981, 9, 3-23.