Update: 2 March 2009. I am withdrawing this post, in the sense that I no longer think it is strictly accurate. See this post to see why. But I want to leave this post up for others to see how not to do statistics.
Right after the close of the Republican and Democrat conventions (in 2008), I asked readers to participate in a study where they could guess who would win the presidential election. They could also indicate who they wanted to win, and if they ordinarily skewed Conservative or Liberal.
The idea was to search for traces of wishcasting, which is what happens when people let their desires influence their judgment about what will happen.
A wishcasting sales manager might suppose his will make a higher sales number next month than he probably will because he wants to save his job. A climate activist might forecast a higher probability of doom because he wants mankind to be responsible for various real and imagined ills. A sports fan will similarly put too much weight on a victory for his favorite team.
Or a voter might guess too high a probability for his candidate’s victory given that he desires him to win.
It is important to recognize wishcasting traits in yourself—or in your company—so that you can reduce or eliminate them and thus produce more accurate, and thus more valuable, forecasts. Recognizing wishcasting in your forecasts is the first step to making better predictions.
Please remember that these guesses were made right after the conventions, well before the end-game of the campaign. At that point, the information on the candidates was roughly equal. Both media reports and personal conversations indicated that both candidates had a chance to win. That would obviously change later in the campaign, but a crude parity was in place after the conventions.
We received 624 legitimate responses (see the Data Caveats section for more details). 79% thought McCain would win, 21% Obama. Obama won 53% of the popular vote, McCain 46%. Given the information they had at the time, voters guessed too high for McCain and too low for Obama.
Not surprisingly, given the nature of this blog, 83% wanted McCain to win, 15% wanted Obama, and 2% chose not say. This was the same breakdown for philosophy: 83% reported Conservative, 15% Liberal, and 2% unknown.
80% of the participants were men. The median age was 51 (same for men and women), with a range between 16 and 89.
The tricky part of this analysis is that it is impossible to say exactly how much wishcasting was done. It might be true that each of the respondents did not let their desires influence their guess at all. Given what we know of human nature, this is probably false, but we cannot say with certainty. Everything below, therefore, is just an estimate.
We also cannot say about the level of wishcasting in any person: all estimates can only be for the “average” voter, that is, the type of voter who would take part in an internet study like this. If we had repeated guesses from one person, then we could hone in on his wishcasting component, but we only have one guess per person, so we can only say something about the group.
This table gives a first indication. It estimates the probability of who wins given who you wanted to win:
Of those who wanted McCain to win, 89% thought he would win, and only 11% thought Obama would. Likewise, of those who wanted Obama to win, 75% thought he would. Of the small number who didn’t express a desire, two-thirds thought McCain would win. That 89% and 75% are pretty big and indicate that some kind of wishcasting has taken place. Why?
If there were no wishcasting, and assuming the 600-some people’s actual votes did not influence the election unduly (presumably everybody who wanted a candidate to win voted for him), we would expect that who you wanted to win would have no bearing on who you thought would win. People should be able to separate their desire from their judgment.
Since overall 79% thought McCain would win, if no wishcasting took place, we would expect that 79% of those who wanted McCain to win to say he would. But 89% did. That 10-percentage point difference is the amount of wishcasting that took place among McCain supporters.
Since overall 21% thought Obama would win, then if no wishcasting took place, we would expect that 21% of those who wanted Obama to say he would. But 75% did. Thus, the 54-percentage point difference is the amount of wishcasting among his supporters.
For the people who didn’t say who they wanted, we would have expected a 50/50 split, but we saw a slight leaning towards McCain, 66/34. We shouldn’t read too much into this, as this was only for 9 people.
In other words, Obama supporters, as a group, wishcasted at a rate five times higher than McCain supporters. This result is not now surprising, given how the election played out, particularly in the media.
We cannot dismiss the element of doomcasting, either. This is when people guess what they don’t want to happen. Doomcasters have a counter-balancing effect on wishcasters, pulling the data back to where we would expect it had people not let their desires influence their guesses in a positive direction. Since we see a wishcasting effect, we cannot reliably estimate any doomcasting effect from the above table.
But we can say more. The next tables are just like the first, but broken up by those who identified themselves as Conservative or Liberal.
Estimated probability of who wins given who you want to win by philosophy:
|Who win?||Who win?|
|Want win?||McCain||89%||11%||Want win?||McCain||89%||11%|
For Conservatives, the amount of wishcasting for McCain was the same: a 10-percentage point bias. But for Obama supporters, the wishcasting fell 10-percentage points to only a 44-percentage point bias.
For Liberals, the amount of wishcasting for McCain was the same. But for Obama supporters, it increased a slight amount, to a 56-percentage point bias.
There were not enough people who did not specify a desired candidate to break down the numbers into philosophical groups.
In other words, Liberal Obama supporters wishcasted at a rate higher than Conservative Obama supporters. The amount of McCain wishcasting remained the same regardless of philosophy, indicating more consistency. These results are also not surprising.
If we break the data down by age into two groups (tables not shown): greater or lesser than the median 51 years, we find the McCain wishcasting bias remains the same, but for the older Obama supporters it goes up to a 65-percentage point bias. This was somewhat surprising given that we heard during the campaign that younger people who preferred Obama were more zealous. The younger group had just about the same percentage point bias as before (49 points).
Among males, the McCain wishcasting bias stayed the same. But among females it increased just slightly to 13-percentage points. The amount of Obama wishcasting bias was the same for both males and females.
There was not enough data to reliably break the data down any further: for example, age by philosophy, sex by age, and so on.
Wishcasting almost surely took place in the McCain-Obama presidential election. This conclusion is conditional on the poll giving usable data (as to that, see the next section).
We must remember that these results are relevant for people who would come to a blog like this, during the specified time, and for elections like the one we had. At the least, we have said something about our small community; at the most, we have said something about average web browsers. We have probably not said much about the general population.
We should understand that any given Obama voter should not have necessarily subtracted 50-percentage points from his guess that Obama would win. That 50-percentage point bias was for the group and not necessarily for any individual. The implies that some people should have subtracted more, some less. The only way to say something about an individual is to collect more data on that individual so that we can estimate his typical bias.
Who wishcasted? Well, your author took part in the study: I thought McCain would win and also wanted him to. Wishcasting was probably there to some extent.
Other McCain supporters wishcasted, too, but not by very much on average. It didn’t seem to matter if they were young or old, male or female, or whether they identified themselves as Conservative or Liberal, the amount of McCain bias was about the same. Again, this was characteristic of readers of this blog, so we should be careful to say the same is true for all McCain supporters.
Those who wanted Obama to win really let that desire influence their guesses. Liberal Obama supporters wishcasted the most on average, a 56-percentage point bias; those who listed themselves as Conservative were more temperate, but not as tempered as McCain supporters. Sex didn’t make any difference to the results, but age did: older Obama supporters wishcasted more than younger ones did, a finding that goes against conventional wisdom. Once more, this result is for readers of this blog, or people like them.
Suppose you are, or were, an Obama supporter and you say, “So what if I say I wanted him to win. He did win, didn’t he? What does wishcasting have to do with anything? I made the right guess.” Yes, you did. This time.
The result—at the time of the conventions—was by no means a forgone conclusion. McCain might have won. You were making a guess about an uncertain future. You got it right this time, but you might not get it right next time. In making predictions of this type, if you are like the typical Obama supporter in this study, you are letting your desires influence your judgment too much, and over the course of many of guesses you will make more mistakes than the folks who do not wishcast. You’ll be losing either money, or prestige, or something.
There are probably plenty of formal studies, of which I am not aware, that show that the more controversial the matter the more people let their desires influence their predictions of the future. These findings do support the anecdotal perception that Obama supporters were more emotional than were McCain voters. It’s not that McCain supporters did not let their feelings influence them—again this data can say nothing about any individual, just the average behavior about a group of persons—but that influence was not sharp.
Obama supporters were, as everybody knows, more passionate (particularly read the next section). They let this passion sway them more. It worked out for them this time, but that passion can easily work against them the next time they make a prediction.
All volunteer-based surveys run the risk of being as valuable as that which comes out the backside of a horse. You never know what you are going to get or who will try and trip you up. So it was with this survey.
In the hopes of getting clean results, I did announce, “Please do not try and stuff the ballot box by voting more than once. I will not release any results until after the election is over. (I will also remove duplicate records.)” However, there is no way to tell if or how many people lied.
I had about 600 entries, most of which went McCain’s way—not surprising given the traffic of this blog. When—suddenly!—a throng of Obama supporters showed up. I was pleased until I saw in this new data some Your Mama fans, along with a good chunk of Spiderman and Batman voters.
What had happened was a faithful reader asked the far left-wing writer of Pharyngula to post the poll. Which would have been okay, but the author called me “right wing”, which sent his impressionable readers into a tizzy. You can go and look for yourself, but they said things like:
Posted by: VegeBrain | September 9, 2008 10:34 AM
I decided to be just as honest as our current commander in thief and
voted in this poll in order to add fuel to the worst fears of
conservatives. So I voted like this:
Who will win the election? Obama
Who do you want to win? Obama
Select your majority political opinion: Conservative
Select your sex: Female
What is your age (nearest year)? 62
It is my honest hope that the conservatives stay awake at night
worrying about retirement age women in their ranks that want Obama to
win the election.
If anyone here can think up an even better combination of choices to
make the Conservatives even more paranoid, please do so.
Posted by: Chris | September 9, 2008 10:12 AM
Well, I participated in the poll, and it is now INVALID, because I am
I love skewing nonsense polls!
Much was worse, despite my posting there and asking them to remain civil because this was an honest study, the spoiling of which would do no good because the results were not going to be released until after the election. My pleas were ignored or were abused.
I also goofed when I used an easily-spoofed HTML form where anybody could deviate from the planned responses. So these people, extraordinarily upset that people were not supporting their candidate, went nuts and injected a few thousand bogus entries into my database. There were many instances of the exact same entries from the same IPs dozens of times all in row, for example. Plus negative ages, enormous ages (one was 2147483647). Strange sexes, fictional political parties, and other junk.
Luckily, I knew the point at which the poll appeared on the radical site, and I was able to exclude all the data from that point on. This means, of course, that I had to throw out some undoubtedly legitimate data as well. I have learned my lesson about this kind of internet poll.