Try to think of an answer before reading further.
Recall that all statements of logic, which includes probability statements, are conditional on stated evidence. As a consequence of this, not all statements have an exact truth value, and so not all have a probability.
Example: What is truth/probability of this statement:
S = “This sentence is false” ?
If S is true (has probability 1), then it is true that “this sentence is false,” therefore the sentence is not true. But if S is false (has probability 0), then it is false that “this sentence is false,” therefore it is true. The circle is complete!
Self-referentiality in math and logic is always dangerous, but it has led to profound insights. Gödel anyone? Is self-referentiality present in the best statistics question ever (BSQE)? Before we come to that, let’s examine this multiple-choice question:
- Who wrote the play Hamlet?
- A: William Shakespeare
- B: PG Wodehouse
- C: John Milton
- D: William Shakespeare
The correct answer is A or D. Further, there is no universal rule that the right answer has to appear only once, though I believe all would agree that it should appear at least once. We therefore assume that the right answer appears at least once in the BSQE.
Finally, Jaynes has shown us (in his magisterial Probability Theory: The Logic of Science) that if any ambiguity in the wording exists, in particular around the phrase “at random”, then the same statement can have two or more different probabilities depending on how the wording is interpreted (example).
Be wary, because any or all three of these complications might apply to the BSQE.
The English phrase, “If you choose an answer to this question at random” might mean, “If you were to chose an answer to this question at random,” or it could mean, “You must chose an answer to this question at random.” If the latter, then the BSQE is self-defeating because requiring a randomly picked answer forces your hand, so to speak. The selection of the answer is beyond your control, which is what “at random” means. Two coin flips (say) determine the outcome.
There does not appear to be ambiguity in “at random”, incidentally. There is no information beyond the information that there are four choices and only one must be picked. This (via Williams’s statistical syllogism) implies a “uniform” or maximum-entropy distribution for picking.
Anyway, to avoid the self-negation, the BSQE could be interpreted to mean, “If you were to suppose another student chose an answer to this question at random (but you must think it through and select one of A through D), what is the chance this other student is correct?”
We now have a problem: how many right answers are there? Is the BSQE structured like the Hamlet question? There is a similarity. The BSQE also has two identical answers, thus on the assumption that there is at least one right answer, A and D are either both right or they are either both wrong. B and C are different, so again they both cannot be right, though both can be wrong.
It is an assumption, though, that because the labels A and D are identical that if one is right both are. That assumption might not hold. Instead, the writer of the BSQE could have said “D is right and none other.” If this is so, then the chance the other student guesses correctly, given our evidence and assumptions, is 25%. And this is also so if the BSQE writer has said, “A is right and none other” or B or D. So if the BSQE writer meant only one answer was right and none other, then the chance the other student guessed correctly is 25%.
But you are not the other student. You must pick only one of A through D. We know the chance the other student guesses correctly is 25%, so you feel free to choose A or D. Choose A. If A was what the BSQE writer picked as the sole right answer, then you are fine. But if the sole right answer was chosen as D, then you are wrong, even though the label on A is the same as D. It would be as if you answered A for the Hamlet question but the teacher marked you wrong because he decided—and it was his choice—that only D was correct. Obviously, the same goes if you pick A but the BSQE writer picked D.
Then we are forgetting that the BSQE writer could have picked C or D. If so, you must choose C or D to have the right answer, even though the labels on these questions have no seeming bearing or relevancy. Again, it would be as if the Hamlet writer said obstinately decided that only C was right.
Thus, if the assumption there is only one right answer is correct, the BSQE implies A or D is correct (for you), but it might be that neither is. Self-referentiality has struck! No clear solution exists.
So change the assumption back and suppose that if A is correct then so is D. They have the same labels, after all. If the correct answer is B, then the probability the other student guesses correctly is 25%, and the same if the correct answer is C. If the correct answer is A or D, then the correct answer is guessed by the other student 50% of the time. We again must assume that the BSQE writer chose a right answer, which could be B or C or it could be either of A or D.
Equipped with this deduction (and it is a deduction), you answer B. If the BSQE writer chose B, then you are sitting pretty. But if he chose A/D or C, then you’re wrong. The same argument as last time applies for the other choices, and thus we are back deep into self-referentiality, the true vicious circle.
It’s here that we wade into deep kimchi. We skated around the question of how the BSQE writer chose his right answer last time, because we had no knowledge which unique letter he picked or how he picked it. To us, his choice was uniform, i.e. random. But supposing A/D are the same we have more information because of the duplication of labels. Did the BSQE writer constrain himself to all four or only three choices? Suppose just three, and he tacked on answer D as an afterthought (regardless whether A is correct). Or did the BSQE writer refer to your thinking process, assuming you would correctly deduce everything properly, and then pick his answer constrained by this assumption? The other student must still guess randomly in any of these scenarios, incidentally, so nothing changes for his guessing chances.
With the repeated reminder that the probability you deduce depends on the assumptions that you make, and thus that different sets of assumptions can lead to different probabilities, I leave the rest as reader exercises.
1Thanks to Zach Woodmansee, HC, commieBob, and an anonymous person. Thanks to John Cook for pointing out a stupid typo.