I was asked to talk about how to handle uncertainty in forecasts. Below is a rough sketch.
Many of you run or work with models that produce forecasts, guesses of the future. The output of these models make statements like, “Sales in June will be up 15%,” “The number of cars sold in 2011 will be 1.4 million,” or “The percent of people who vote for candidate A will be 38%.”
Typically, these forecasts will in the form of points, single numbers, as the examples suggested. But nobody believes that the sales in June will be exactly 15%, that the number of cars sold will be precisely 1.4 million, and that the number of voters for A will be equal to 38.000%.
When we hear these numbers, we automatically, depending on our experience level, add a “plus or minus” where the width of that adjustment is wider or narrower depending on whether or trust in the forecast is low or high.
The National Weather Service used to be reluctant to issue precipitation forecasts other than to say that it was or wasn’t likely. Today, we hear probabilities such as “The chance of rain is 40%”, which provide a fuller understanding of the uncertainty in the prediction.
Incidentally, since it is often asked, that “40%” is defined to mean that, during the relevant period stated (usually 12 hours), there is a 40% chance that at least a trace of precipitation will fall somewhere in the relevant location (usually a large area). It does not mean that precipitation will certainly fall over 40% of the area.
Stating the forecast probabilistically is necessary for those who make decisions about precipitation. For example, farmers deciding to irrigate, you deciding whether to carry an umbrella. Of course, these probabilities are married to precipitation quantity forecasts (called QPFs in the lingo) which increase their usefulness.
No forecast is complete without some indication of its uncertainty. If you run an economic model whose output is a single point you are making a very strong statement. It is no different than claiming that you are certain that the future is known, that car sales will be 1.4 million.
Adding an internal “plus or minus” moves you away from dogmatism, but the problem is that these adjustments can be swayed by emotion, inexperience, or desire (moving a forecast in the direction you want it to go is called wishcasting; moving the thing forecast in the direction of forecast is called cheating). Also, you cannot know whether your “plus or minus” is the same as the next guy’s.
Obviously, you would like some way to account for uncertainty systematically. Models come in two pieces: inputs and internals. Suppose an economic model which produces forecasts of car sales. Some of its inputs might be outputs from other forecasts, such as GDP, unemployment rate, and so forth. Other inputs might have more specific information on the car industry: this information might be knowns—such as the number of manufacturers—or uncertainties, such as iron prices.
The internals might, and usually are, black boxes, software packages provided by third parties. Even so, all models are sets of equations, algorithms, various modules and parameterizations, and so forth.
The easiest thing to do is to run the model with your inputs fixed. Then, since those inputs are not certain, “perturb” them—change them in accordance with their uncertainty—and then rerun the main model with the perturbed inputs.
For example, suppose our model has one input, a GDP guess for 2011 of $14.2 trillion. Run your model with that as an input. Your car sales model spits out 1.4 million. But GDP will not certainly be $14.2 trillion. It might be $14.1 or $14.3 trillion. Re-rerun your model twice more with each of these as inputs.
Repeat as necessary, once for each possible GDP input. You will have an ensemble of outputs, which you can then weight by the uncertainty of each of its inputs. If you have adequately represented the uncertainty of the inputs, your ensemble will automatically give you a “plus or minus” envelope around car sales.
Of course, that example is somewhat screwy because car sales are part of GDP. Or maybe it’s not so screwy after all, since many economic models that use GDP as input forecast something that is part of GDP.
Again, this was just a rough sketch. We haven’t begun to discuss how to handle the uncertainty inherent in the model’s internals. The trick is in fixing an experiment where both the model’s inputs and internals are perturbed in such a way to fully quantify your uncertainty in the eventual forecast.
We’ll get there!