I made a mistake when I wrote about wishcasting in the McCain-Obama presidential election.
I’d like to thank Patrick Hadley for keeping up his criticisms which led to me to see more clearly how to make the correct calculations. Mike D and JH also contributed, and Luis Dias put his finger on the problem. And I’d like to apologize to readers for misleading them.
The mistake I made can be illustrated with an example.
Let’s first remember that all probability is conditional on information or evidence. For example, Idice = “This is a die, just 1 side is labeled 6” and if we want to calculate the probability of seeing a 6 on a throw, we have to condition on the evidence Idice. Thus
Pr(See 6 | Idice) = 1/6,
and Pr(See “not 6” | Idice) = 5/6, where “not 6” means 1, or 2, etc.
Now suppose I ran a poll where I asked a bunch of people “What number do you think will show on the next roll?” It turns out there are two groups in my bunch: regular guys off the street (GOS) and degenerate gamblers. Both groups naturally know Idice and should base their guess on that information. We should, therefore, see the following results if both groups used the information:
The probability of seeing an outcome given a group:
| What shows | |||
| Not 6 | 6 | ||
| Want show? | GOS | 83% | 17% |
| Gamblers | 83% | 17% | |
That is, regardless of which group we consider, 83% of its members should guess a number 1-5, and 17% should guess 6, if they used Idice. (We can continue adding as many groups as we like, but they should all break down 83%/17%.)
But suppose when we ran the poll, we saw this table:
| What shows | |||
| Not 6 | 6 | ||
| Want show? | GOS | 83% | 17% |
| Gamblers | 50% | 50% | |
Something has gone wrong with the gamblers. They have, as a group, evidently wishcasted the 6, to the tune of 50% – 17% = 33% bias.
The way we arrived at this number is the same way I arrived at the McCain/Obama numbers.
McCain-Obama
We can now see that to calculate the amount of group wishcasting requires knowing the conditioning information, like the Idice. What did I use for the McCain/Obama information? I effectively used I79/21 = “The probability of McCain winning is 79%” because that was the overall vote in the sample we had (for why I used that, see below).
This information averages over both the McCain and Obama groups and perhaps does not adequately account for the amount of wishcasting in each group. I say “perhaps” because, of course, I79/21 might be correct. Why?
It would be better to write the information as Iconvention=”All the relevant information available immediately after, but not too long after, the convention.”
Now, if I79/21 = Icon, then everything reported Friday was correct. But it can be argued, persuasively I think, that Icon would imply something less extreme than the 79%/21% breakdown. How much less extreme?
I can remember having conversations at that time with Obama supporters (and in New York City where I live, that’s about all there were), and many at that time thought that McCain would win. I recall hearing a gloomy lecture from two docs. One said, “I think race will be a factor in this election.” I said, “Yes, many people will vote for Obama because he’s black.” They took the opposite view. Anyway, point is, these two button-wearing, rally-attending Obama supporters thought McCain would win. They were not atypical. Many people at the time of the conventions thought McCain had a good chance. That would all change, of course, as the campaign played out. Those two docs, for example, would later “forget” they had ever said McCain would win.
So what was the exact Icon? I don’t know, and nobody else does, either. But here is a graph of various possibilities. The x-axis shows the probability of a McCain victory implied by Icon: Obama’s probability is 100% minus this, of course. The red dots show the amount of wishcasting bias on the y-axis in the McCain group; the blue dots show the amount of wishcasting bias for the Obama supporters.
The vertical red line is at the sample we got, 79% (21% for Obama). It shows that the McCain group had a 10% wishcasting bias, and the Obama group had a 50% bias.
The blue line is at the 50%/50% split. The McCain people had a 40% bias at that point, and Obama group had 25% bias. But the 50%/50% split is probably too low in favor of McCain—at the point of the end of the Republican convention when this poll took place (see below about national polls).
The black line is at the point where the two groups had about the same amount of bias: about 32% at the 57%/43% split. That point might be, I think it can be argued, a reasonable interpretation of Icon.
There are two points where no wishcasting bias would be present in the two groups: I89%/11% would give 0% bias for the McCain group, and I25%/75% would give 0% bias for the Obama group. I don’t think either information set can be defended.
So, overall, both groups wishcasted, and both probably did so at the same amount. We can’t tell for certain because we can’t know or reliably estimate Icon: it is a matter of interpretation (I can well imagine my two doc friends now saying something Icon = I5%/95%; whereas at the time they were estimating Icon = I60%/40%).
What stands?
The breakdowns by age, philosophy, and sex are still accurate in terms of direction or non-movement of the wishcasting bias. For example, Liberal Obama supporters, regardless of Icon, still had a larger bias than Conservative Obama supporters. As did older over younger Obama supporters. It is only the exact bias numbers that are uncertain.
How did it happen?
I wanted to get the results out quickly, and I was lazy. Plus, using the observed “marginal” I79/21 is standard practice in calculations of independence, so I used it and forgot about it. Then I got distracted by my real job and by some criticisms—well meaning criticisms—that were wrong, and I figured if they were wrong I must right. Not a good use of logic, that. This has taught me to take my time in the future.
A criticism that is wrong is that “the sample was skewed by McCain supporters.” That is false because it does not matter how many McCain supporters I had, as long as they were not wishcasting, just like in the gamblers/non-gamblers example. But given that some of the McCain supporters were wishcasting, then I could not disentangle how much they were, or how much the Obama people were, or if either group used information other than I79/21.
Some other criticisms focused on the fact that it not possible to separate desire from prediction. However, this is certainly possible. Polls were mentioned. But polls are surveys where the question, “Who do you want to win” is asked, and not “Who do you think will win?” So you cannot back out wishcasting from a poll, nor use its results because of that.
Normally, one calculates wishcasting by taking a series of forecasts from one person, or groups; but it’s the series that counts. Because if you have a series you can use it to back out what the information I should have been. Since you can estimate the information, you can estimate the wishcasting.
I will try, if I can find the time, and set up a series of polls, for non-political topics, to show how this can be done. Say, that gives me a good idea…
Thank you for the acknowledgement. I hope that I am not too strident and rude in my criticisms, but I did find it really odd that you made this mistake, and that it was not noticed immediately by everyone.
Are you being a little ironic when you say in your update “I am withdrawing this post in the sense that I no longer think it is strictly accurate”? Strictly accurate? The post was completely invalid, it was a false and inaccurate use of statistics and probability.
You say that using the observed marginal in calculations of independence is standard practice, fair enough but do statisticians ever argue: “IF A THEN B” when they know that A is false and then go on to use the B figure as if A were true? That is always fallacious has to be a recipe for disaster whenever it is used.
Patrick,
I mean “strictly” in its logical sense. That is, if the information Icon = I79%/21% is correct, then all the findings in the post are correct. It is unlikely I79%/21% is true, but not impossible. To say “completely invalid” means that you have deductive proof to show that I79%/21% is in fact false, not just unlikely.
Also, as I pointed out today, all the statistics and probability calculations I gave were absolutely correct and accurate given I79%/21% was true.
The method is correct regardless whether I79%/21% is true or false.
To show no ironic intent, here is what I said on that post in its entirety:
Matt and Patrick:
Perhaps this is a type of parable from which we can all learn: Humility requires tough-mindedness.
The key part of the method that I criticised as invalid, false and inaccurate was the calculation of I79%/21%. That was where the method went wrong; once you have done that all that remains is subtraction. To average out the expectations and assume that this is what would have happened had there been no wishcasting is the main part of the method and is certainly wrong.
If by some chance in the real world the neutral expectation happened to be 79%/21% then obviously all you would have to do would be compare this to the expectations found in the survey. You would get the same answer but that would certainly not justify your method. The answer might be right on rare occasions – but that does not mean the method is correct.
If Icon= I79%/29% is false then the method gives the wrong answer. A method that gives a wrong answer is not usually described as correct.
The method is wrong regardless whether Icon=I79%/21% is true or false.
No, my dear Patrick, you are wrong.
A method can certainly be correct but be fed incorrect information. This does not make the method wrong, but the result wrong. That is what I admitted to.
The only point in dispute is the conditioning information (the 79%/21% split). If that was right, then everything I did was right. If it was wrong, which I have admitted is probable, then the results are wrong. However, the method itself is sound, applicable, and correctly used.
I have even supplied the answers for different conditioning information. You are free to pick the point which you believe represented the information right after the convention. I gave one argument for something like the 57%/43% split; however, I am certainly willing to entertain others.
The gamblers example should certainly convince you of the soundness of the method.
“A criticism that is wrong is that “the sample was skewed by McCain supporters.†That is false because it does not matter how many McCain supporters I had, as long as they were not wishcasting, just like in the gamblers/non-gamblers example”
But McCain supporters were clearly wishcasting, so I was correct. I didn’t state it as technically or mathmatically when I said
“Second, the main reason 79% overall thought McCain would win in your poll is because the majority of the people who took your survey wanted McCain to win, and 89% of those thought he would win. If you had an even representation of people who wanted Obama and McCain to win (50%-50%) the expected outcome would be the same… 89% of McCain Wanters think McCain will win and 75% of Obama Wanters think Obama will win.”
but i think that basically says I79/21 was the wrong Icon
Dr Briggs I hope you do not mind me continuing to discuss this point. I am not being simply awkward or trollish, I enjoy visiting your site and have learned a lot from it.
I think the reason why we did not agree about the merits of your method is that we were talking about different things.
By your “method” I meant the method you used in the McCain/Obama example. You took a weighted average of the expectations of the two groups, said that if there had been no wishcasting the expectation of each group would have been equal to that average; and then subtracted the expectation of each group from the weighted average.
In the gamblers example you did not take a weighted average of the expectations of the guy in the street and the gamblers; nor did you not compare the expectations of each group with that weighted average; instead you just subtracted the expectations of the gamblers from what was known a priori to be the correct value for the expectation. (Since some “guys in the street” may not be quite so au fait with the laws of probability it is best to use the known result rather than rely on a survey.)
It seems that what you are talking about when you say “the method” is simply subtraction of the expectation from the true figure.
If I am correct about this then there is no need to continue this discussion any further, but I hope you can understand why I thought that you were talking about something more than simple subtraction when you said that the method was sound.
I assumed I-con was 50/50, which I still think is the most logical predictive supposition. But altering that assumption to the (weakly supported) I-con = 57/43, then the McCain wishcast rate was 89-57 = 32% and the Obama wishcast rate was 75-43 = 32%.
Equally dreamy!!!!!
Which, I suspect, is why you chose 57/43 (fess up,now). The magnanimous conclusion: the propensity to substitute wishes for cold logic taints us all pretty much equally. A peacemaking gesture, in other words.
However, equality in wishcasting propensity is not the case. Some of us (McCain-supporting doomcasters, 11% of the sample) saw beyond our wishes to the stark reality of the situation. We put aside our hopes and dreams and evaluated the situation with emotionless scientific clarity.
Here in the aftermath, as the nation slides deeper into social and economic collapse, with absolutely the wrong people at the helm, it might behoove the dreamy masses to consult with the realists. However, realist that I am, I predict that is never going to happen. In the land of the blind, the one-eyed man is feared and shunned.
To Mike D, …”In the land of the blind, the one-eyed man is feared and shunned.” …and lobotemized. Recall the “Invasion of the Body Snatchers.” It’s taken 50 years to find out, but the pods did win.
Mike,
It’s funny that you said, “Here in the aftermath, as the nation slides deeper into social and economic collapse, with absolutely the wrong people at the helm, it might behoove the dreamy masses to consult with the realists.” One of my good friends who’s a policy wonk in DC said the exact opposite.
This is why I try to stay out of politics as much as possible and stick to discussing more important topics. Like “Battlestar Galactica” and “The Office.”
I pointed it out too, zoomie moron. And I can’t beleive you made this kind of mistake. Basic surveying one oh one should have given you the insight. Not sure you have your head completely straight yet either.
Sad, really.
The thing you can tell from your survey is that someone is wishcasting. You can’t even tell which group is wishcasting for sure, since the neutral expectation might be outside the bounds of eithe the Obama or McCain group.
You also can’t use your poll numbers as indicators of the expected election, since you don’t know the general size of the sample groups within the general population. and since Tradesports is a better predictor than polls anyhow.
This whole thing was a mess.
I’m really NOT interested in getting the book now. Think the only decent thing in there would be the cartoons.
I wonder what you do at work. Wonder how good that book is. Wonder how often you get yourself confused with a bunch of formula dancing instead of thinking.
Oh…and you wishcasted like a biyatch in the last 2 weeks of the election. Predicted a McCain win despite polls against it, Tradesports with huge odds against it, etc. And you were wrong. What a joke. How much money did you lose at Tradesports? If you really beleived the bullshit you were writing, you should have put a bunch down. You had great odds, considering your even money expectation of McCain likely winning.
Oh…and Joy needs to eat some crow too. Or maybe if she still things the original analysis was good, WM can tell her why she’s wrong. Of course….at this point, I don’t trust WM. Sad.
TCO,
You’re not a very happy person, are you, TCO. I can’t help but speculate on the cause. Maternal neglect? Vitamin deficiency? Pure mean-spirited cussedness, perhaps?
I will say I am man enough to admit when I’m wrong, and do so in just as loud and clear a voice as when I think I’m right. You have the same record?
P.S. You know the rules. One of your comments had to go. Resubmit it after correcting, if you like.
P.P.S. Don’t forget that all probability is conditional on certain information. The information that led to the “John McCain will win” post was nothing—nothing—but the historical Democrat vs. Republican elections. Shame on you for implying otherwise.
1. Actually I don’t know the rules. No cussing? Will follow.
2. You did admit you were wrong. Kudos.
3. I still gasp that you made the error. I think it shows you getting caught up in equations and losing “physical insight” (physics term). Good analysts need both equation manipulation and insight. And you really needed a lot of “head beating on” to get you to see your fallacy. With all respect, it worries me, that this was the case. Wonder where else you pull boners. Wonder if you are even wary enough of your own proclivity to logical mistakes.
4. I’ll go and dig up your election comment, but I think it was more than just noting that statistic and making a pointless hypothetical. That you said you personally thought McCain would win (either endorsing that statistic, or encompassing other intuitions). And Tradesports had him down 90-10 at that point. Why you would bring up some stat like the one mentioned for insight instead of a relevant pointer…like eh…Tradesports, polls, etc. boggles my mind. I mean you might as well bring up the fff…oolish “Redskins” metric.
5. I am sort of an angry little man. You can find pictures of my aspects at the Flame Warriors site. Don’t shrink my head though. And, please disaggregate both my angry tone and lazy writing style from the basic points I make.
6. I think you’re actually kind of nice and intersting and such. Still…messing up this example…and going to Heartland and not recognizing what losers most of those fellows are…worries me.
7. Yes…I am very good at admitting strongly when wrong. I have a lot of other bad tendancies. But I’m good at this one. (You asked.)
Ok…I looked up the post I was referring to, regarding your personal McCain wishcasting. It was at 1834 on 06OCT08. It was NOT in the context of that silly statistic. It was your stated expectation.
Tradesports had it 80-20 or 85-15 or so. Not 90-10. See? correcting myself. Am too lazy to calculate though.
Now, TCO, my little angry friend. You know that when I was invited to last year’s Heartland conference, I gave the identical paper I had just given (two month’s previously) at the Annual Meeting of the American Meteorological Society. My paper on hurricanes likely not increasing in strength or number was such a hit, I was not invited back for this year’s conference (Heartland, that is).
As for the McCain/Obama wishcastings: everybody makes mistakes.
I just think it’s funny how clubby the denialists are (and I’m a skeptic btw…I have just learned to hate a lot of my own side for wishcasting.) I mean Watts is full of kaka…and McIntyre totally holds his fire. Same thing with Loehle…even Douglas. It’s pretty obvious that these guys are not total lay it on the line, say the truth about your own brother kind of guys.
TCO,
I’ve read and re-read the posts, the argument and my comments and I still think I’m right. I won’t be drawn on what Briggs or you or anyone else said or why they said it. That is the truth; and I certainly don’t need any more explanation.
I won’t be eating a crow or any other garden bird. Strictly chicken or turkey for Joy. No daffy, Donald, Matilda, and I hope Disney doesn’t invent a story about a chicken or a turkey any time soon as I’ll run out of options. Irrational? Yes.
When I think I’m wrong you will know.
What does concern me though is that you are so furious about nothing, and I’m not being sarcy.
It’s not for you to write however and whatever you like and to expect others to pick the bones out. It is for you to construct full rounded sentences that exactly express your thoughts coloured by how you feel if necessary. This is always a challenge. It is enough to be gentle.
As for Heartland conference, which starts tomorrow, I believe your comment is really aimed at Steven M. I’m just sad that it’s not in London as was suggested.
Maybe Briggs can explain to you the error. See what you caused, WM? Got ninnies like this continuing to defend your (pretty fundamental) boner. This is what makes me sad about the denialists. They think with their hearts. They’re not willing to shoot their own dogs. Robert Heinlein would not love them. Lack ability to face facts or insights contrary to prejudice. The alarmists tend to be similar. But I expect better from denialitsts…since they claim revisionism.
TCO,
If Robert heinlein shot his dog I wouldn’t love him either!who ever he is.
And denial isn’t just a river in Egypt
Eat your crow, Joy. Be brave.
I was wrong, TCO,
I meant Mortimer, Arabel’s pet magpie who used to sort through the coal scuttle every day to checkit for diamonds to add to his collection.
It was peer reviewed and published in 1980. I didn’t mean Matilda. That will teach me to write in such a hurry! We all learn from our mistakes.
Raven, not magpie, there I go again!
http://blog.wired.com/geekdad/2008/06/the-geekly-read.html
Briggs had a math/logic flaw. He has admitted it. You were wrong to support him. Now you refuse to admit your error.
TCO,
Read my comments, you, like others, clearly did not. I can find no fault with what I wrote.
Bully tactics and creepy crowing will get you nowhere.
When you can think straight yourself, you have a real name, are polite and you still think you have a point to make I’ll be all ears.
There are men fighting and dying on this globe, Joy. And you can’t admit an error. Sad.