This is a bit of a preview of a paper my friend Russ Zaretzki are working on.
Take a gander at his pic:
This is the probability that John McCain wins the election given only the historical evidence of Republican/Democrat elections, and the fact that there will be just 1, 2, …, up to 38 more Republican/Democrat elections. Let me explain.
Since Democrat James Buchanan ran against Republican John C. Fremont in 1857, United States presidential elections have been dominated by these two parties. From that first contest, Democrats have won 16 elections and Republicans 22. This year we have another election in which the two parties are again featured. Now, this means that the number of elections of this type has so far been finite, and history strongly suggests that this series of elections itself will be finite; that is, some day it will not be Democrats versus Republicans, or even might even be that there will be no elections1.
How many more elections there will be is, of course, an open question. But let us suppose that the one before us is the last election between the two parties. Then, conditional only on the past elections, the probability that the victor will be a Republican is 0.577. The standard Bayesian (continuous-value approximation) estimate gives 0.575. The classical guess is 0.579.
Our new method of guessing is based on knowing that the number of elections has been and will continue to be finite, that is, that it will not be without number, going on forever. It is important to recognize that traditional methods make this assumption. That is, that the number of “trials” (elections) will be infinite.
Ok, ok. These don’t seem like very big differences—and for this problem, they are not. But let’s suppose that instead of this being the final election, we’ll have two more. Then the probability McCain wins is just over 0.575. If we think there will be 9 more elections, then the probability McCain wins this one is only 0.570. Once the number of future elections becomes “large”, our guess matches the standard Bayesian one. That’s what the dashed, black horizontal line is. The red dot-dashed line is the classical estimate.
Eh, not a very big difference either, but it could be enough of one if you were, say, making a bet. And in some other problems, the differences are enormous; but this problem is a lot more fun.
The probability is over 50%. It obviously does not account for anything except previous elections. But it’s enough to raise a smile.
Incidentally, the math for all this is very heavily related to Laplace’s probability of succession. Google that. We introduce a twist that makes solving it sensible for certain problems. The surprise is that the probability depends on knowing the future number of trials (that’s the big difference).
——————————–
1 Ever notice that at the Democrat rallies you hear “Obama! Obama! Obama!”, while at the Republican ones you hear “USA! USA! USA!”?
Here is the latest Gallup chart of polling results from October 6-8 to date:
http://www.gallup.com/poll/111568/Gallup-Daily-Presidential-Race-Narrows-Slightly.aspx?version=print
Not so clear cut after all. You may still collect on your bets Mr. Briggs!
Don’t miss Iowahawk on Balls and Urns.
http://iowahawk.typepad.com/iowahawk/2008/10/balls-and-urns.html
-Steve
Uh oh. I think you need to write another book to explain all this.
I don’t know HOW I got here by following a link from a Wiki article about “Rumpole of the Bailey” which is infuriating and I will be reporting. But here’s a few facts to put into your historical equation you might have forgotten.
1. The Republicans were the out and out Liberals when they started. For example: The Republican Candidate Abraham Lincoln (you may have heard of him) was considered *dangerously* liberal. That sort of why States started seceding from the union. That means they didn’t want to be in the United States anymore if the technical language confuses you.
2. When the Republicans started playing by a conservative party line they started started this thing called the Depression which bears a marked resemblance to the current economic crisis. Learning from mistakes doesn’t seem to be a big thing on their agenda.
3. And please, that graph up there only shows how desperate Republicans are to prove that McCain is going to pull some kind of election rabbit out of the electoral hat. And this is a guy who picked a running mate who is so ultra conservative to the point that she has a teen aged daughter who is pregnant, is given a $150,000 wardrobe that garnered so much bad press they gave it BACK, and can’t even hold her own with Katie Couric. What a GREAT presidential decision maker McCain is, I can’t tell you!
P.S. I want to make a point that this comment is NOT exactly because i am a Democrat, I tend to keep my political decisions to myself and leave others to theirs. BUT YOU HIJACKED A FREE ENCYCLOPEDIA OF KNOWLEDGE AND TRIVIA FOR A POLITICAL STATEMENT THAT IN A FEW DAYS IS NOT GOING TO MEAN ANYTHING ONE WAY OR ANOTHER! YOU JUST *HAD* TO EMPTY YOUR ALIMENTARY TRACT ALL OVER A SITE THAT IS DEDICATED TO THE FREE SHARING OF KNOWLEDGE WHETHER IT BE TRIVIAL OR NOT. NEXT TIME YOU WANT TO DO SOMETHING LIKE THIS FIND AN OUTHOUSE!!!
Thank you for playing but next time bring you brain,
Deb Terry
Deb,
I note you made your post at midnight. The witching hour. I think you might need more sleep.
“Hijacked”, my dear? Strong word. Means to take by violent force. However, it might interest you to know that I did not make that Wikipedia entry, and had no idea of its existence. It’s probably there because for years I ran a “Rumpole of the Bailey” web page (for proof, go to the Way Back machine). I took it down about two months ago because it received almost no traffic.
Your point, if I can summarize it, seems to be that any method that estimates a reasonable probability of success of a Republican victory is not just wrong, but immoral. Thus, you have given us a wonderful example of why people should vote Republican. Thank you.
SteveBrooklineMA,
That link is great. His conclusion that the actual poll error is what’s important, and to hell with the theoretical error is right on.
Mike,
In a couple of days we’ll post the article to arxiv and I’ll give a link here.
Deb:
Was Abraham Lincoln an American President?
I think you’re casting nasturtiums about Briggs.
On Palin’s wardrobe:
Take a look at men’s versus women’s clothing. You will find that with little exception women’s are priced higher. Even a product made of the same fibre and with the same workmanship. Women will pay. Men will not. Women are vain and will pay to look better. Men know they are the most handsome beast that has ever walked God’s green earth and do not need to pay to look better.
How much were Obama’s suits? Did we hear about this? He is very dapper in them. Palin’s suits are to be given to charity. How many guys will want to buy Obama’s old suits? The Dems seem to fear Palin’s looks as an unfair advantage and were spitting feathers when McCain’s running mate turned out to be a looker. Who is more likely to vote with their eyes? The women, for Obama/Biden, or the men, for McCain/Palin.
On Wikipedia:
If, when using it, one has to think first whether the subject is likely to be spun for any reason, i.e. it involves human endeavour: history, inventions, wars, politics, religion, hockey sticks, guns, who’s dictionary is best, which side you should spread your butter on a Rivita (they have a war about that in re-enactment circles), whether to put the milk in first in a cup of tea and all things pink and fluffy, bright and beautiful; then I should take everything with a grain of salt, or simply view the information as the opinion of the publisher. Wiki’s not workable as a font of truth.
Nothing man made is free without added extras excepting thought, feeling and physiology- Joy’s law.
So, you can stop with the violins about Wikipedia’s virtue and if you’re unhappy about something you can go and change it back to what ever you want. It’s a battle of wills, money and patience. Great concept, not fit for purpose in reality without prior knowledge of the subject in question.
On Mr Briggs:
Watch and learn, he pulls a rabbit out of the hat most days of the week. But he doesn’t have a cape, he is not shy, thinks latex is the way forward and wants to be Emperor!
Heaven help us.
More concern about poll error:
http://www.washingtonpost.com/wp-dyn/content/article/2008/10/28/AR2008102803675_pf.html
The last paragraph in the article – “He noted that to address “one potential pitfall,” The Post and ABC conduct interviews with a random selection of those who have only cellular phone service alongside a traditional random sample of those with residential phone service. One recent criticism of current polling has been that it does not accurately capture the sentiments of those who primarily use cellphones.” By Michael Abramowitz, Washington Post Staff Writer, Wednesday, October 29, 2008
I wonder if they take into consideration those users who have their cellphones set on vibrate as opposed to ringtone…
Great links. I especially like the first one and the catchy catch phrase, “guilty of aggravated bullshit in the first degree.” I’d post that on the fridge but my wife would remove it so I won’t bother. She doesn’t like that kind of thing but lucky for me tolerates many of my foibles because I am, as noted, a handsome beast.
btw, Laplace used for his example of the rule of succession calculation of the probability that the sun will rise tomorrow. One of my favorite stat puzzles. His result is not completely agreed upon either. Logic is not always logical, and great minds do not always think alike.
Mike,
Yeah, Laplace has taken more grief for that example than any other mathematician ever. I’d say it was his bad choice of the sunrise that set probability back 100 years.
The idea is sound. Given only the information that the sun has rose so many times, you can calculate the probability it will rise tomorrow. Problem is, nobody, not no body, can ever keep in mine just the information “the sun rose this many times so far.”
They start criticizing the information. “It wasn’t that many times, it was this.” “It ignores what I know about this or that” and on and on. None of those objections are in the least interesting. None change the probability of rising tomorrow given just the fact “the sun rose this many times so far.”
It is a lesson that choosing the right example means the world.
Bruce,
Good link. I mean, someday soon, to talk about Rumsfeld’s “unknowns unknowns”, a statement that he took a lot of grief over, but it is course perfectly accurate.
This formula only works if the subject test is independant of outside(non-probabilty) influence. I fail to see the US presidential election as a %chance event. If it were, you wouldn’t see parties winning elections in clusters of 8-12 years at a time, high frequency of incumbant candidate re-election etc. The distribution does not follow the expected outcome of a random 58% probability event.
You can choose to ignore [“It wasn’t that many times, it was this.†“It ignores what I know about this or that†and on and on.] and sure the formula give you the %chance occurance, but it’s only true if the data input is actually %chance data which wasn’t generated in part or in whole by the outside factors you are ignoring.
John,
If you say that there is additional evidence to consider when calculating the probability, then we agree. All probability statements are conditional, don’t forget. So the above calculations are correct on the given evidence and not on other evidence.
I guess my point was, given your parameters, the math exercise is valid yet irrelevant.
John,
Oh, no. Not irrelevant. In fact, it is a bit of a surprise that the probability of a victory depends on how many more R/D elections we will see in the future. This dependence will be true even if you add more relevant information.
Now, to math geeks like me, this is kind of cool.
A problem I am having with this “number of future elections will alter the probability of McCain winning” is this –
The outcome of the 2008 election will have been determined before the number of future elections is known. Right now there is a secret, armed to the teeth, underground military organization waiting to overthrow the US government, and once they act they will win and never let go. If they act in 2010 or if they wait untill 2110 it will have no bearing on the 2008 election. Unknown future events do not retroactively alter past probability.
As long as the number of future elections remains unknown, the true probability is 0.575. If we are told before the election X more elections will be held, things could change.
Please provide an explaination or link why I am wrong. Or for that matter, some explaination of how the graph was generated. I know it’s under review blah blah; it sounds interesting but tough to swallow.
John,
Best references are all of Chapter 3 of Jaynes. (Bibtex refs are below) A great historical paper is by Zabell. And you can’t go wrong with the classic Diaconis and Freedman Annals papers.
Your thinking is right on if probability were a physical thing. But it is not. It is a measure of logic and not of causality. If you knew information from the future (such as the number of D/R elections that will be) then that of course changes the information you have about this upcoming election. And since probability is a measure of information/logic, we can compute its probability.
You are right on the money where you say if we don’t know the future number of elections, but suspect they will be large in number, then the probability is 0.575.
Actually, I owe you thanks. I have been wondering why our curve exhibits the structure it does: starts hi, swoops low, asymptotes to the standard Bayesian estimate. I understood the last part, but not the first. But now I have it! It’s because the expectation over the future values must average out to the asymptotic Bayesian answer. So of course some of the estimates must be higher and some lower than the Bayesian answer: if some weren’t higher and lower, the average would never make it! (Sounds like gibberish, I know, but the math will show it. I’m quite happy about this revelation.)
@Article{ DiaFre1980,
title = “Finite exchangeable sequences”,
author = “Persi Diaconis and David Freedman”,
journal = “Annals of Probability”,
pages = “745–764”,
volume = “8”,
number = “4”,
year = “1980”
}
@Book{ Jay2003,
AUTHOR = “E. T. Jaynes”,
TITLE = “Probability Theory: The Logic of Science”,
PUBLISHER = “Cambridge University Press”,
YEAR = “2003”,
address = “Cambridge”
}
@Article{ Zab1989,
title = “The rule of succession”,
author = “Sandy L. Zabell”,
journal = “Erkenntnis”,
volume = “31”,
year = “1989”,
pages = “283–321”
}
I told you that you would need to write another book to explain it.
at mccain rallies, they cheer “nobama! nobama! nobama!”
http://www.youtube.com/watch?v=vL20TdHjX2s
Considering the coin toss example, are you saying that knowing how many more times I would like to toss the coins will affect the probability of current toss?
I do think John’s point is valid. The intuition that unknown future events do not retroactively alter past probability could undermine your surprise.
Regarding your answer to his question, are you saying that you have put a prior distribution on the “number of future elections†in your calculations? Anyway, I can also wait to read the paper to find out the answer.
JH,
The coin tossing example is different because you have other logical evidence that says that probability of any flip is 1/2. Conditional on that evidence, the probability of the next is 1/2.
The election is not like a coin flip but is more like the hackneyed balls & urns (in our paper, we inventively turn the urn into a box: that’ll show ’em!).
To prove to yourself that future events can logical change the current probability, imagine an urn with 20 balls, half black, half white. You have removed none as yet, but you know that Jane will, two weeks from now, remove 10 and that all of them will be black. Now, what is the probability that the ball you draw two seconds from now will be black?
Well, that example is simple. Jaynes has many more, but they quickly become more mathematical. Also see his Chapter 6.
“1 Ever notice that at the Democrat rallies you hear “Obama! Obama! Obama!â€, while at the Republican ones you hear “USA! USA! USA!â€?”
it goes even further than that… my wife and i attended the Palin Rally here in LA (Carson actually) last month, as did a small number of Obama supporters camped at the entrance. when a USA USA chant started, the Obamatons retorted with “No USA! No USA!”.
pretty telling, no?
JH,
No, the logical probability in the coin toss example is 1/2 no matter what. The coin toss example has extra known information that you are conditioning on.