Skip to content
February 27, 2009 | 36 Comments

Wishcasting the McCain-Obama presidential election

Update: 2 March 2009. I am withdrawing this post, in the sense that I no longer think it is strictly accurate. See this post to see why. But I want to leave this post up for others to see how not to do statistics.

Background

Right after the close of the Republican and Democrat conventions (in 2008), I asked readers to participate in a study where they could guess who would win the presidential election. They could also indicate who they wanted to win, and if they ordinarily skewed Conservative or Liberal.

The idea was to search for traces of wishcasting, which is what happens when people let their desires influence their judgment about what will happen.

A wishcasting sales manager might suppose his will make a higher sales number next month than he probably will because he wants to save his job. A climate activist might forecast a higher probability of doom because he wants mankind to be responsible for various real and imagined ills. A sports fan will similarly put too much weight on a victory for his favorite team.

Or a voter might guess too high a probability for his candidate’s victory given that he desires him to win.

It is important to recognize wishcasting traits in yourself—or in your company—so that you can reduce or eliminate them and thus produce more accurate, and thus more valuable, forecasts. Recognizing wishcasting in your forecasts is the first step to making better predictions.

Results

Please remember that these guesses were made right after the conventions, well before the end-game of the campaign. At that point, the information on the candidates was roughly equal. Both media reports and personal conversations indicated that both candidates had a chance to win. That would obviously change later in the campaign, but a crude parity was in place after the conventions.

We received 624 legitimate responses (see the Data Caveats section for more details). 79% thought McCain would win, 21% Obama. Obama won 53% of the popular vote, McCain 46%. Given the information they had at the time, voters guessed too high for McCain and too low for Obama.

Not surprisingly, given the nature of this blog, 83% wanted McCain to win, 15% wanted Obama, and 2% chose not say. This was the same breakdown for philosophy: 83% reported Conservative, 15% Liberal, and 2% unknown.

80% of the participants were men. The median age was 51 (same for men and women), with a range between 16 and 89.

The tricky part of this analysis is that it is impossible to say exactly how much wishcasting was done. It might be true that each of the respondents did not let their desires influence their guess at all. Given what we know of human nature, this is probably false, but we cannot say with certainty. Everything below, therefore, is just an estimate.

We also cannot say about the level of wishcasting in any person: all estimates can only be for the “average” voter, that is, the type of voter who would take part in an internet study like this. If we had repeated guesses from one person, then we could hone in on his wishcasting component, but we only have one guess per person, so we can only say something about the group.

This table gives a first indication. It estimates the probability of who wins given who you wanted to win:

Who win?
McCain Obama
Want win?    McCain 89% 11%
Obama 25% 75%
None 67% 33%

Of those who wanted McCain to win, 89% thought he would win, and only 11% thought Obama would. Likewise, of those who wanted Obama to win, 75% thought he would. Of the small number who didn’t express a desire, two-thirds thought McCain would win. That 89% and 75% are pretty big and indicate that some kind of wishcasting has taken place. Why?

If there were no wishcasting, and assuming the 600-some people’s actual votes did not influence the election unduly (presumably everybody who wanted a candidate to win voted for him), we would expect that who you wanted to win would have no bearing on who you thought would win. People should be able to separate their desire from their judgment.

Since overall 79% thought McCain would win, if no wishcasting took place, we would expect that 79% of those who wanted McCain to win to say he would. But 89% did. That 10-percentage point difference is the amount of wishcasting that took place among McCain supporters.

Since overall 21% thought Obama would win, then if no wishcasting took place, we would expect that 21% of those who wanted Obama to say he would. But 75% did. Thus, the 54-percentage point difference is the amount of wishcasting among his supporters.

For the people who didn’t say who they wanted, we would have expected a 50/50 split, but we saw a slight leaning towards McCain, 66/34. We shouldn’t read too much into this, as this was only for 9 people.

First finding

In other words, Obama supporters, as a group, wishcasted at a rate five times higher than McCain supporters. This result is not now surprising, given how the election played out, particularly in the media.

We cannot dismiss the element of doomcasting, either. This is when people guess what they don’t want to happen. Doomcasters have a counter-balancing effect on wishcasters, pulling the data back to where we would expect it had people not let their desires influence their guesses in a positive direction. Since we see a wishcasting effect, we cannot reliably estimate any doomcasting effect from the above table.

But we can say more. The next tables are just like the first, but broken up by those who identified themselves as Conservative or Liberal.

Estimated probability of who wins given who you want to win by philosophy:

  Conservative          Liberal
Who win?        Who win?
McCain Obama        McCain Obama
Want win?    McCain 89% 11%        Want win?    McCain 89% 11%
Obama 35% 65%        Obama 23% 77%

For Conservatives, the amount of wishcasting for McCain was the same: a 10-percentage point bias. But for Obama supporters, the wishcasting fell 10-percentage points to only a 44-percentage point bias.

For Liberals, the amount of wishcasting for McCain was the same. But for Obama supporters, it increased a slight amount, to a 56-percentage point bias.

There were not enough people who did not specify a desired candidate to break down the numbers into philosophical groups.

Second finding

In other words, Liberal Obama supporters wishcasted at a rate higher than Conservative Obama supporters. The amount of McCain wishcasting remained the same regardless of philosophy, indicating more consistency. These results are also not surprising.

Third finding

If we break the data down by age into two groups (tables not shown): greater or lesser than the median 51 years, we find the McCain wishcasting bias remains the same, but for the older Obama supporters it goes up to a 65-percentage point bias. This was somewhat surprising given that we heard during the campaign that younger people who preferred Obama were more zealous. The younger group had just about the same percentage point bias as before (49 points).

Fourth finding

Among males, the McCain wishcasting bias stayed the same. But among females it increased just slightly to 13-percentage points. The amount of Obama wishcasting bias was the same for both males and females.

There was not enough data to reliably break the data down any further: for example, age by philosophy, sex by age, and so on.

Conclusion

Wishcasting almost surely took place in the McCain-Obama presidential election. This conclusion is conditional on the poll giving usable data (as to that, see the next section).

We must remember that these results are relevant for people who would come to a blog like this, during the specified time, and for elections like the one we had. At the least, we have said something about our small community; at the most, we have said something about average web browsers. We have probably not said much about the general population.

We should understand that any given Obama voter should not have necessarily subtracted 50-percentage points from his guess that Obama would win. That 50-percentage point bias was for the group and not necessarily for any individual. The implies that some people should have subtracted more, some less. The only way to say something about an individual is to collect more data on that individual so that we can estimate his typical bias.

Who wishcasted? Well, your author took part in the study: I thought McCain would win and also wanted him to. Wishcasting was probably there to some extent.

Other McCain supporters wishcasted, too, but not by very much on average. It didn’t seem to matter if they were young or old, male or female, or whether they identified themselves as Conservative or Liberal, the amount of McCain bias was about the same. Again, this was characteristic of readers of this blog, so we should be careful to say the same is true for all McCain supporters.

Those who wanted Obama to win really let that desire influence their guesses. Liberal Obama supporters wishcasted the most on average, a 56-percentage point bias; those who listed themselves as Conservative were more temperate, but not as tempered as McCain supporters. Sex didn’t make any difference to the results, but age did: older Obama supporters wishcasted more than younger ones did, a finding that goes against conventional wisdom. Once more, this result is for readers of this blog, or people like them.

Suppose you are, or were, an Obama supporter and you say, “So what if I say I wanted him to win. He did win, didn’t he? What does wishcasting have to do with anything? I made the right guess.” Yes, you did. This time.

The result—at the time of the conventions—was by no means a forgone conclusion. McCain might have won. You were making a guess about an uncertain future. You got it right this time, but you might not get it right next time. In making predictions of this type, if you are like the typical Obama supporter in this study, you are letting your desires influence your judgment too much, and over the course of many of guesses you will make more mistakes than the folks who do not wishcast. You’ll be losing either money, or prestige, or something.

There are probably plenty of formal studies, of which I am not aware, that show that the more controversial the matter the more people let their desires influence their predictions of the future. These findings do support the anecdotal perception that Obama supporters were more emotional than were McCain voters. It’s not that McCain supporters did not let their feelings influence them—again this data can say nothing about any individual, just the average behavior about a group of persons—but that influence was not sharp.

Obama supporters were, as everybody knows, more passionate (particularly read the next section). They let this passion sway them more. It worked out for them this time, but that passion can easily work against them the next time they make a prediction.

————————————————-
Data Caveats
Continue reading “Wishcasting the McCain-Obama presidential election”

February 25, 2009 | 30 Comments

Peer review

Here is how peer review roughly works.

An author sends a paper to a journal. An editor nearly always sends the paper on to two or more referees. The referees read the paper with varying degrees of closeness, and then send a written recommendation to the editor saying “publish” or “do not publish.” The editor can either accept or ignore the referees’ recommendations.

The paper is then either published, or sent back to the author for revisions or rejection.

If the paper is rejected, the author will usually submit it to another journal, where the peer review process begins anew. This cycle continues until either the paper is published somewhere (the most typical outcome) or the author tires and quits.

Here are two false statements:

(A) All peer-reviewed, published papers are correct in their findings.

(B) All papers that have been rejected1 by peer review are incorrect in their findings.

These statements are also false if you add “in/by the most prestigious journals” to them. (A) and (B) are false in every field, too, including, of course, climatology.

A climatology activist might argue, “Given what I know about science, this peer-reviewed paper contains correct findings.” This is not a valid argument because (A) is true: the climatology paper might have findings which are false.

If the activist instead argued, “Given what I know, this peer-reviewed paper probably contains correct findings” he will have come to a rational, inductive conclusion.

But a working climatologist (gastroenterologist, chemist, etc., etc.) will most likely argue, “Given my experience, this peer-reviewed paper has a non-zero chance to contain correct findings.” Which is nothing more than a restatement of (A).

The “non-zero chance” will be modified to suit his knowledge of the journal and the authors of the paper. For some papers, the chance of correct findings will be judged high, but for most papers, the chance of correct findings will be judged middling, and for a few it will be judged low as a worm’s belly.

Here is a sampling of evidence for that claim.

(1) Rothwell and Martyn (abstract and paper) examined referees’ reports from a prominent neuroscience journal and found that referee agreement was about 50%. That is, there is no consensus in neurology.

(2) No formal study (that I am aware of) has done the same for climatology, but personal experience suggests it is similar there. That is, there is at least one published paper on which the referees do not agree (at what is considered the best journal, Journal of Climate).

(3) Pharmacologist David Horrobin has written a commentary on peer-review in which he argues that the process has actually slowed down research in some fields. He also agrees with my summary:

Peer review is central to the organization of modern science. The peer-review process for submitted manuscripts is a crucial determinant of what sees the light of day in a particular journal. Fortunately, it is less effective in blocking publication completely; there are so many journals that most even modestly competent studies will be published provided that the authors are determined enough. The publication might not be in a prestigious journal, but at least it will get into print.

(4) I have just received an email “Invitation to a Symposium on Peer Reviewing” which, in part, reads:

Only 8% members of the Scientific Research Society agreed that “peer review works well as it is”. (Chubin and Hackett, 1990; p.192).

“A recent U.S. Supreme Court decision and an analysis of the peer review system substantiate complaints about this fundamental aspect of scientific research.” (Horrobin, 2001)

Horrobin concludes that peer review “is a non-validated charade whose processes generate results little better than does chance.” (Horrobin, 2001). This has been statistically proven and reported by an increasing number of journal editors.

But, “Peer Review is one of the sacred pillars of the scientific edifice” (Goodstein, 2000), it is a necessary condition in quality assurance for Scientific/Engineering publications, and “Peer Review is central to the organization of modern science…why not apply scientific [and engineering] methods to the peer review process” (Horrobin, 2001).

This is the purpose of the International Symposium on Peer Reviewing: ISPR (http://www.ICTconfer.org/ispr) being organized in the context of The 3rd International Conference on Knowledge Generation, Communication and Management: KGCM 2009 (http://www.ICTconfer.org/kgcm), which will be held on July 10-13, 2009, in Orlando, Florida, USA.

Be sure to visit the first link for more information.

(5) Then there is the Sokal Hoax, where a physicist sent a paper full of gibberish to a preeminent social science journal to see if it would be published. It was. Sokal was careful to play to the preconceptions of the journals’ editors to gain acceptance. The lesson is the oldest: people—even scientists!—easily believe what they want to.

(5) John Ioannidis and colleagues in their article “Why Current Publication Practices May Distort Science.” The authors liken acceptance of papers in journals to winning bids in auctions: sometimes the winner pays too much and the results aren’t worth as much as everybody thinks. A review of the article here.

(7) UPDATE. Then there is arxiv.org, the repository of non-peer-reviewed “preprints” (papers not yet printed in a journal). Arxiv is an acknowledgment by physicists, and lately mathematicians and even climatologists, that it’s better to take your findings directly to your audience and bypass the slow and error-prone refereeing process.

(8) It is easy to get a paper into print when the subject is “hot”, or when you are friends with the editor or he owes you a favor, or your findings shame the editor’s enemies, or through a mistake, or by laziness of the referees, or in a journal with a reputation for sloppiness. In most fields, there are at least 100 monthly/quarterly journals. Thus it is exceedingly rare for a paper not to find a home, no matter how appalling or ridiculous its conclusions.

Update: 4 March; Reader Jack Mosevich reminds us to see this article at Climate Audit. Real-life example of politically correct refereeing.

The listing of these facts is solely to prove that (A) and (B) are false, and that peer review is a crude sifter of truth.

Thus, when an activist or inactivist points to a peer-reviewed paper and says, “See!”, he should not be surprised when his audience is unpersuaded. He should never argue that some finding must be true because it came from a peer-reviewed paper.

This web page has also tracked several peer-reviewed, published papers that are crap. Examples here, here, here, and here (more are coming).

————————————

1Incidentally, I have only had one methods paper rejected; all others I wrote were accepted to the first journal I sent them to. Nearly every collaborative paper I co-wrote has also been accepted eventually. I am an Associate editor at Monthly Weather Review, and have been a referee more times than I can remember, both for journals and grants.

I mention these things to show that I am familiar with the process and that I am not a disgruntled author seeking to impugn a system that has treated him unfairly. To the contrary, I have been lucky, and have had a better experience than most.

February 24, 2009 | 15 Comments

I suppose Apple—corporation—would be happy

Sigh. I wonder why these Enlightened souls bother with school. They have already told themselves that they know everything there is worth knowing.

The only other thing worth noting—the video speaks, loudly, for itself—is that I recall reading that many of the students emailed their professors asking to be excused from assignments, tests, and the like while they were holed up. Never reported was what their professors replied.

February 23, 2009 | 25 Comments

What appeal to authority means and what it doesn’t

This article is meant to be the first is a small series of demonstrations of how not and how to argue for or against climate activism. The level of argumentation on the web has long passed subbasement (people have been calling each other “Hitler” for over two years), but worse are the misuses and misunderstandings of logic. People throw around terms like ad hominem and appeal to authority constantly, without understanding what they are saying.

My hope is that when you see an abuse of the type outlined, you simply cut and paste the link to these pages. This will save us all a lot of time and unnecessary typing. I hope.

1.

A prominent climate inactivist forwarded me a document in which he argued against some of the more catastrophic claims said to be due to global warming.

At the beginning of his piece he implied that James Hansen, who is the best known climate activist, should not be trusted because Hansen only had training as an astrophysicist and not as a climatologist.

This is a poor argument because the author of the piece was not himself a climatologist. If you must be an official climatologist before being allowed to comment on climatology—a position that is logically valid given that you can satisfactorily define “climatologist”—then just about everybody, activist and inactivist, must shut up. Including the author of that piece—and almost certainly, including you.

2.

Mr Activist: this means that you would not be allowed to say anything whatsoever about global warming except to repeat what you have been told by an official climatologist. You would be allowed to say “Mr Climatologist says B” and nothing else.

(“B” can be any statement or proposition about climatology only—it cannot be about politics or health or biology or anything else.)

Mr Activist would not be allowed to say “Mr Climatologist says B, and you’re a fool not to believe it.” Pause and understand this. The reason is that he is not qualified to say what and what is not foolish because he is not a climatologist.

If an official climatologist says “B, and people would be fools not to believe it,” then you can repeat that statement. But you cannot adorn it, nor comment further, nor say anything else. You can repeat what you are told by your betters and then you must keep quiet.

The only exception to this logical rule is that if everybody, activist and inactivist, agreed on the additional premise, “People that do not believe what official climatologists say are fools.” Then you can logically say, “Mr Climatologist says B, and you’re a fool not to believe it.”

But almost certainly, everybody would not agree on that premise. Let’s see why.

3.

The prominent inactivist made a logical mistake by arguing “Because Hansen is not a climatologist, his statements on climatology are false.” This is only valid if only climatologists make true statements about climatology and if non-climatologists always make false statements about climatology. Is that true?

Obviously not. Plenty of climatologists have made statements about climatology that turned out false in fact and in theory. And plenty of non-climatologists have made statements about climatology that turned out true in fact and in theory.

Because of this, it necessarily means that anybody is allowed to say anything they want about statements of fact or theory about climatology. This includes both activists and inactivists. We can now see that neither side can accuse the other of making a logical mistake by talking about climatology.

So stop arguing about this point!

4.

Suppose Hansen is an official climatologist and he makes the statement, H = “The global average temperature in 2010 will be at least half a degree hotter than in 2009.” (Whenever we see ‘H’, we must remember that it stands for “The global average…”)

If an activist then says, “Hansen, an expert, says H. Therefore, because Hansen is an expert, H is true.” This argument is invalid; the activist has made a mistake. H cannot be true because Hansen said so. The logical error made is called “appealing to authority.”

Most know of this mistake and avoid obvious instances of it. But see 5.

Suppose a second activist said, “Hansen, an expert, says H. Therefore, because Hansen is an expert, H is likely to be true.” This is not a mistake and is a rational thing to say. This is because experts making statements like H are often, but not always, right. Therefore it is rational to suppose that the expert is likely to be right again.

The statement made by the second activist is an appeal to authority, too, but a sound, inductive one.

Mr Inactivist: it does you no credit to accuse non-climatologists of being irrational if they are making arguments of the second type. It is often wise to appeal to authority like this, and is what we all do when we enter an aircraft, trusting the pilot to get us safely to our destination.

5.

Most will agree that Hansen meets the definition of climatologist, but then so does your author, and so do several people (like Dr Lindzen) who do not always agree with what Hansen says.

Now we have trouble. For if only official climatologists can make true statements about climatology, and if two (or more) official climatologists make contradictory statements about climatology, then we have a logical contradiction if the premise “Only climatologists make true statements about climatology” is true. We have already seen it is false, so we are safe.

But suppose Mr Activist says, “Most climatologists say B. Therefore B is true.” This is the same logical error: appeal to authority in the deductive sense.

Let the second activist say “Most climatologists say B. Therefore B is likely to be true.” This is a perfectly rational thing to say.

Even more, the first premise appears to be true in fact: Most climatologists do agree on most statements B about climatology. Therefore, it is rational for people, and their close cousins politicians, to say to themselves “B is likely to be true.”

Mr Inactivist: your only appeal, if you believe B to be false, is to marshal arguments against B. You must not call B a “hoax” or use other disparaging terms as you risk being guilty of appealing strictly to your own authorities (however, there is more to say here, but we’ll save this for another time).

Mr Activist: Because you have reasoned B is more likely to be true, you cannot say “Therefore, everybody must believe B.” To do so is to make the same appeal-to-authority logical error. You must also not express amazement that anybody dare disagree with B for the same reason, and because you must remember that the inactivist has consulted his own authority or set of facts and is arguing inductively just as you are.

6.

Mr Activist, you must not say that Mr Inactivist’s authorities are ineligible because they do not agree with your authorities. This is the same logical error: appeal to authority once more.

And it is a foolish thing to say because you risk defining “expert” solely as “somebody who agrees with what I want.” That is not a logical error, but it is asinine.

7.

Finally, Mr Activist, you must understand Mr Inactivist is making an inductive, and therefore rational, appeal to authority, when he argues “Yes, most climatologists say B, but I believe they are mistaken because these other climatologists claim to have shown where the first are in error.”

Thus, if Mr Inactivist says, “Therefore, B is likely to be false”, then he has said a rational thing. But if he has said, “Therefore, B is certainly false”, then he has made the same error and you can call him on it.

8.

Update: I want to leave as an exercise about how arguments about “peer review” and “consensus” fit into the appeal to authority arguments. After you have said something about “peer review”, then read this article.