This is a tad incoherent, but the gist is here. I had the opportunity of submitting an abstract to the AGU fall meeting, and had only a couple of hours in which to do it. This is the, eh, plain-English rendering of that abstract. Stand by for more news.
People trying to escape the implication of a bad forecast often claim their forecast wasn’t a forecast but a projection or scenario. The implication is that a bad forecast means a (possibly beloved) theory is no good. Therefore, if the forecast wasn’t a forecast, but a projection or scenario, the theory can still be admired (or funded).
This won’t do. Forecasts are scenarios are projections. And bad forecast-scenario-projections means bad theories.
These misunderstandings are not only found in making predictions, and in classifying which future-statements count as predictions, but also under which circumstances predictions must be verified. There is general recognition that good models produce good forecasts, but bad forecasts can’t be ignored by calling them a projection or scenario.
Now the finer points. For a start, the remarks below are general and apply to any data not yet seen, but for ease, predictions of future events are illustrated.
All forecasts are conditional on two things: a theory/model and a guess about what the future holds. Neither need be quantified or even rigorously defined, of course, but since scientists are keen on quantification, models usually have numbers attached to them.
Imagine the simplest model, which is a function of the past data, of time, and some set of premises which specify the model form (say, an ARMA process). This model can make a forecast. It will be conditional on the theory—which is the past data and model-form premises—and on a guess about what the future holds—which here is just that the future will come at us in discrete time points, t+1, t+2, etc.
Suppose the forecast is for time points t+17 and t+18. Here you stand at t+3, well short of t+19, but you still want to “verify” the forecast. Well, you can’t. The guess of the future has not obtained, therefore the forecast hasn’t, in effect, really been made. It is null. It is impossible to discuss the quality of the theory: it may be good or bad, we can’t know which.
Nothing changes if we add to the model other propositions of interest. Suppose we augment our simple time series model with “x” variables, propositions which, for the sake of argument, say something about matters probative of the thing forecasted. Now if the forecast does not change in any way regardless of the state of the “x” propositions, then these items are irrelevant to the theory. Irrelevant items shouldn’t even be part of a theory, but these days, in this heyday of the politicization of science, anything goes.
For illustration, add another component to “x”, say, the price of oil exceeding some level. Point is this. If the guess of the future is t+1 and t+2, and here you stand at t+3, you have met the time criteria, but still have to check whether the price of oil exceeds the stated level. If it does, you can check the validity of the forecast; if not, not.
Nothing changes if we add other “x” variables, or turn the model into physics instead of statistics. Get it? There is no difference between a physics and statistics models in terms of forecasts. (Most models are mixtures of both anyway.) If the guess of the future conditions obtains, then the forecast may, even must, be evaluated (the technical term is “verified”).
A difficulty arises with the word scenario, which is “overloaded”. It can mean guess of what the future holds, the time points plus the price of oil, i.e. the “x”, and is therefore not a forecast but part of one, or it might mean a forecast. To avoid confusion, this is why only “forecast” or “prediction” should be used.
It’s time t+3 and the price of oil did not exceed the stated level, so the forecast is null. But, since the price of oil is probative, we could make a new (after-the-fact) forecast assuming the appropriate price. In this way, we can still verify the model.
If we’re using that model for making decisions, particularly in government, we must verify it. We must input the “scenarios”, i.e. the “x”s that obtained, and then recompute the forecast. If the forecast has no skill, the model must be acknowledged as unworthy, to be abandoned of overhauled.
Update Reading is a difficult art, rarely mastered. Many after reading a title feel they have assimilated all the material there under. Strange. Many others gloss. More than a few shot right by where I said scenarios were sometimes the “x” and sometimes the forecasts themselves.
But it sure is nice to have an opinion, isn’t it?