— TheRightStuff (@ThaRightStuff) November 3, 2016
Some crow is presumably being eaten by Nate Silver and the folks at FiveThirtyEight.com this fine morning. I recommend they bake it into a pie (which I learnt from an episode of Rumpole).
This isn’t the first time Silver & Crew have got it wrong about Trump. Indeed, his group is on a failure streak. To his credit, Silver admitted his mistakes, writing earlier “How I Acted Like A Pundit And Screwed Up On Donald Trump“. I did the same thing with Obama—twice (which makes me dumber). I, like Silver, let wishcasting get the best of me.
And, like Silver, though he doesn’t know it yet, I relied on a bad model—not once, but twice. Mea maxima culpa. Silver is still relying on his model, which performed well before, but which I believe is now misleading him.
The first time Obama ran, Silver created a quantitative model which took account of information others had not, or had treated less than optimally. Silver’s New & Improved! quantitative model made better predictions than other models, and thus he very rightly was rewarded (that he told our elites what they wanted to hear is beside the point).
I have predicted Trump’s victory since January (and as regular readers know) have believed it months before that. Silver has long been predicting Trump’s defeat (just as he jokingly predicted the Cubs’ loss back in May). But Silver and I are using different models, which therefore give us different probabilities. So first a lesson in probability (which you can learn all about in this must-have book).
Probability does not exist. Therefore nothing has a probability. It is always an error to say things like “The probability the Cubs win is X”, “Trump’s chances are Y.” By “always” I mean always.
Since probability does not exist, and nothing has a probability, the only way to speak of probability is in reference to exterior information. You can say, “If you believe the following things, the probability the Cubs win is X” and “According to my model, Trump’s chances are Y.”
What’s a model?. A model is a set of propositions that are, or are assumed, true. That’s it and nothing more. The propositions don’t even have to be related to one another: they can be gibberish. But gibberish is not going to make for a good model.
Models always speak of a proposition of interest, such as “Cubs win”, “Trump wins”. The goal of modeling is to find a set of premises (propositions) that are probative of the proposition of interest such that the probability deduced from the model is high or one (or low and zero). Actually, it’s more than that. Premises which make the POI high are easily found. Take “The Cubs will win” as the model. Then the probability “Cubs win” given that model is 1. But this would obviously be a lousy model at the start of the season (it is the true model now).
What we’re after in models are not only probative premises, but premises which are true, or are believed true, at the time the model is created. What were the best premises back in May which gave “Cubs win” high probability? Well, we can argue about that. What were the best premises back in January which gave “Trump wins” a high probability? We can argue about that, too: followers of politics will have lots of examples spring to mind.
Silver had and has (I’m guessing) a fixed set of premises, many or which are strictly mathematical, and which tacitly assume that this election would be “much like” past elections. Those premises include the data from various polls (data are always premises); those premises also include information on how to manipulate that data. It’s all very rigorous and scientific.
It’s the rigor and sciency nature of the mathematics which misleads many. They look upon the equations, shiny and beautiful—and they are—and they become alive. The Deadly Sin of Reification strikes! The model becomes—in the mind of the user—reality itself. Before he sins, the modeler says “Given my model, the probability Trump wins is X”, but after Reification hits, the modeler says, “The probability Trump wins is X”. Probability has been conjured into existence!
But it’s all a fantasy.
My guess (and it’s only a guess) is that the rigor and past good performance of Silver’s model has dissuaded him from realizing many of the premises in it are now false. This election is not like others. The polls mean different things than they have before. The labels (and that is all they are) “Democrat” and “Republican” are amorphous. If these changes are not made in his model—and, truthfully, it’s a mystery how to best incorporate them—his model will make rotten predictions. And Silver & Crew have been making bad predictions.
Now my model is not quantitative. Probability is usually not a number, anyway. My model does say, without quantifying, “Trump is likely to win”. If he loses, then I’ll have to examine the individual premises which made the model and discover which were in error.
But even if he wins, it’s not clear the premises I used will be as efficacious in the future. It is not even necessarily true that my model had true or good premises! I could have got lucky.
Predicting complex events is hard (anything to do with human behavior is complex). Much, much harder than those who tout “machine” or “deep learning” algorithms or whatever the going term of the day has it.
Incidentally, it is not true that a set of premises (that we can know) exist which allow us to make perfect predictions. Think, for example, of quantum events. There we can prove we cannot know the premises which say what will happen with certainty. There is no anti-Hari-Seldon like equivalent proof for human events, but I believe the same limitations hold.