Would you check the results of a model with another model? Before you answer, be sure you know what the question is.
A model—whether it is physical, statistical, mathematical, or some combination—is an algorithmic device designed to make predictions about some observable thing. You want today to know that price of tomorrow’s Dow Jones Industrial Index? There are models for that; usually statistical models.
You want today to know whether it will rain in Detroit tomorrow so that you can decide whether to plant your crops in the old lots that used to contain houses? There’s a model for that; a physical-statistical weather model called MOS (model output statistics; see Part II).
Now, how would you, assuming you are not an expert in these matters, check the accuracy of your model? Would you (a) compare the model’s predictions with what actually happened, or (b) produce another model and check the results of the first model against the predictions of the second?
The right answer is (a), of course, but the problem is that there are two ways to interpret “what actually happened.” You probably thought it meant “what happened in the future.” Now, it is the great shame in the field of statistics—both in the dismal way it is taught and the worse way it is practiced by most—that (a) is nearly always is interpreted to mean “what happened in the past.”
Nearly all—the exceptions to this are rarer than sober Paul Krugman columns—statistical models, and many physical models, are checked against the data that was used to fit, or create them. Since it is an elementary theorem that any model may be made to fit perfectly—not just closely, perfectly—to any set of historical data, to claim that your model is good because it fits old data well is a hollow boast.
This is the reason for the great overconfidence of experts who build and use models. And don’t think it doesn’t matter, because it does. People in charge of us makes decisions and set policy based on these models frequently. We are at the mercy of bad statistics.
Weather and Climate Models
But it’s not all bad. It is to the great glory of meteorological models that they are usually—in practice, I mean—checked against what happened in the future. Weather models have the advantage of a constant stream of model predictions and future observations. Discrepancies between the two are noted quickly and used in tweaking the models so that they perform better in the future.
Anybody who cares to look will discover that the performance of meteorological models has improved dramatically over the last thirty years. Of course, people’s expectations of accuracy has also increased, so that the level of grousing about weatherman has remained constant. Human nature.
Climate models are in a different category. So far, all they can boast about is how well they fit the data used to build them, which we have just seen is no great shakes. This being true, those who use climate model output should be humble, they should be cautious, even timid about their prognostications. And that’s just what we see in practice, right?
Actually, it’s still worse, because climate modelers—and in their development stages, weather modelers—answer (b) to that question above. They check their models against the output of other models. How could this be?
Climate/weather models take current observations as input and produce forecasts of future observables as output. But these physical models cannot take observations raw, like statistical models can. They must first process those observations so that they fit into the model environment. This assimilation is called an analysis. Analysis is a model itself.
Climate/weather models are run on grid-like structures, but observations come irregularly: we do not have equally spaced observations over the surface of the Earth and through the atmosphere. To operate, the observations have to be placed on the model grid. The analysis, then, is a sort of interpolation that does this. This is not a detriment; it is a necessary step to get these models to run.
Once the analysis is complete, the model is integrated forward in time to produce a forecast. OK so far? Because it’s about to get tricky. At that future point—the time of the forecast—come new observations. Ideally, the climate/weather model’s output would be checked against these actual observations, at only the irregularly spaced sites where they are taken. These observations are, are the truth, the whole truth, and the only truth.
But that’s not what happens. Instead, these new observations are read into the model in a new analysis cycle. This interpolates these new observations to the model grid. Then the old model integration is checked against this new analysis.
Thus, the model’s accuracy is checked with another model.