Last week we looked at how the polls did in Iowa. How badly did the polls in the New Hampshire race do in yesterday’s Democrat primary? Here are the results:
| Candidate | Zogby | WMUR | Actual | Error |
|---|---|---|---|---|
| Obama | 42 | 39 | 37 | +5 |
| Edwards | 17 | 16 | 17 | 0 |
| Clinton | 29 | 30 | 39 | -10 |
| Others | 5 | 7 | 5 | 0 |
| Undecided | 7 | 8 | 0 | +7 |
| ?????? | ||||
| Huckabee | 10 | 13 | 11 | -1 |
| Romney | 27 | 26 | 32 | -5 |
| McCain | 36 | 31 | 37 | -1 |
| Others | 20 | 21 | 18 | +2 |
| Undecided | 7 | 9 | 0 | +7 |
We still tracked Zogby, but had to switch to the local CNN/WMUR/University of New Hampshire poll for the second comparison.
Last week, for Democrats, the total absolute summed error was 30 points. This week, it is only 22 points, and two candidates, Edwards and “Others”, were predicted perfectly, yet this week’s polls are widely thought to be poorer. Why is this?
The polls last week, while they did not do as well as this week’s in predicting the actual vote percentages, got the order of the top three candidates correct. For this week, the first two candidates were switched, but just barely; the margin of victory was only 2 points.
For Republicans, last week, the total error was 14 points. This week: 16 points! Both last and this week, however, the order of the top three was correctly predicted.
What matters most to people about poll accuracy is the candidate order; the actual predictions of vote percentages don’t seem too important, which is odd. Candidate order is supremely important in a two-person race, of course, where the only thing that counts is who is on top. But when there are multiple choices in a primary and multiple primaries, a good poll must also do well with the percentages—which the polls seem able to do. So don’t lose faith in the polls just yet.
All the polls were taken on the weekend before the primary, of course, and do not, and can not, account for what happens between the polls and the vote. For example, this past week saw Hillary’s breakdown, her interaction with the “Iron my shirt” guys, and Bill Clinton’s stage phone-whisper “I wuv you”, which all happened after the last poll was taken.
This only means that the plus or minus error you often hear reported (“This poll is accurate to within +/- 3 points”) is imperfect, and should in general be larger than given. These plus or minus points are actually theoretical results based on obscure math and many data assumptions, which rarely hold. So it’s far better to use an error rate based on the actual observed error of polls instead of the theoretical intervals.
The mean absolute (Zogby) poll error for the last two weeks, for both parties, was 4 points. This gives some hints that the theoretical error rates are too low and should be, say, doubled. I’ll post more on this after the next polls in Michigan and South Carolina come in.
