This inescapable fact, that the conditioning evidence is only assumed and is therefore subjectively defined, is responsible for much acrimony about accuracy. Thus the reason we attempt to eliminate wiggle room and that we insist on clear lines in advance is obvious. We don’t want some slicker coming along and claiming to be better than he is or right when he was wrong. Just in case it isn’t obvious, let’s pick a non-controversial topic to illustrate the intolerable nature of fuzziness. 
At Fong’s Great Wall restaurant you open your fortune cookie which reads, “You will find happiness.” You pay the bill, head out to the car, and find a quarter on the seat. Does that make you happy? Next day, your learn that the project on which you were to work was canceled. Your workmate takes you out to lunch and foots the bill. Happy? Or perhaps you are a bureaucrat at the Environmental Protection Agency, stuck in a dour cubicle in some sub-basement, daily writing papers on newly discovered menaces and risks to our planet, papers which none but other bureaucrats will ever read. Then, the day after opening the fortune cookie, you learn your department has been cut from the budget and you will finally find freedom. Thus shall you find happiness?
It’s obvious why we should ignore prophecies from Chinese-American pastries: they are too absurd, too loose, too variable to be considered seriously. But how much more soberly should we consider the economist who says, “The Economy (stock market, company, etc.) will improve”, the sociologist who predicts, “Racism (disparity, inequality, etc.) will continue”, the journalist who writes, “The candidate I secretly support (but whom I hope nobody guesses I secretly support) will improve his election chances”, the military strategist who says, “Iran will make aggressive moves”, the environmentalist who says, “The climate will warm and then bad things will happen”, or the futurologist who says anything?
Each of these statements are so ridiculously watery that they fit any receptacle. They are intellectual tea leaves: anything can be read into them. They are all fortune-cookie predictions, except that fortune cookies are superior: at least with the fortune you get a cookie. These statements are free of calories, and they are harmful if swallowed. They are “predictions” that simultaneously obtain or fail, depending on how sympathetic one is to the person making the prediction and not how closely the predictions match empirical observations. These intolerably vague scenarios are one large reason scientists (and other folk) are more confident of themselves than they have a right to be.
In short: each of these are scenarios and not predictions. A scenario has all the trappings of a prediction. It looks like one. It smells like one. It can be used like one. But when you cut it open and examine its viscera, you discover it is a different creature altogether. Scenarios are used by the nefarious to cheat, aggrandize, and bamboozle. They are issued in lieu of predictions by the ignorant. The saddest scenarios are those that not only fool the recipient but also the issuer.
Scenarios are predictions lite. They have conditions, just like predictions do, but one is not meant to look too closely at them. They are meant to be vague, or rather blank, the interpretations to be filled in later. The fortune cookie was a scenario, as are most psychic and political predictions. The scenario’s conditions are infinitely malleable, allowing each individual to find comfort in them, if they are so predisposed.
Saying “the economy will improve” sure sounds like a prediction, doesn’t it? But it’s no different than when a palm reader says “I see difficulties ahead.” For what does “improve” mean? Can it be rigorously defined? What makes up the economy—what specifically? And when will this “economy” improve: tomorrow? next week? next year? Given the heterogeneity of things economic, and the imprecision of dates, it is with a probability approaching certainty that eventually some quantitative measure will show improvement. This new, but unstated proviso, makes it almost certain that the prediction will be a hit.
But the heterogeneity also makes it all but certain that other economic quantitative measure will show a deterioration, which (if taken as a proviso) makes it almost certain the the prediction will be a miss.
As we’ve stressed here before, it is well to be clear that “almost certain” does not mean “certain”. Nor does “practically false” mean “false”. Saying that an event is “practically false” is logically equivalent to saying she is “practically a virgin.”
This took more time than originally planned. The IPCC scenarios are next. Promise!
Meh…so you’re basically saying any so-called prediction that’s too vague to be useful is a scenario. I still like my definitions better.
What you call a scenario, I’d simply call a vague prediction. If I wanted to get technical, maybe I’d say, “fortune telling.”
In my scenario at least three more persons will comment on this thread.
(What do I win? Do I have to pay taxes on it?)
49erDweet: Define ‘person’. 😉
Yes, we will all die. Yes, the economy will improve. Yes, racism will continue. Not predictions. Facts of life.
Matt’s comment (first, at 10 Jan 2012, 7:44 am) seems to cut to the chase — which really appears to be about semantics.
What’s missing, still, after three installments (& counting) is why this definitional parsing is so important? ‘
Very nice metaphor — much better than an allusion to slipperiness — although, just to be pedantic about it, grammatically it should read “All of these…” or perhaps “Each…is…it fits…”
As for “intolerable nature of fuzziness,” that quality is acceptable, even desirable, for baby animals.
Other than that, I’m eagerly awaiting your swat at the IPCC scenarios.
The advice from my boss was to never forecast both a direction or a timeframe.
I’ll allow “both a direction and a timeframe” or “either a direction or a timeframe” but not “both a direction or a timeframe”.
Why is quibbling about words right? It isn’t always but it is when people draw conclusions as though the words were right. If they plan the world economy based on a scenario they thought was a prediction then people will die, indeed, have already died.
@ Will.
JH, Ken, Gary, Doug M and Rich lookalikes are all “persons” under the Geneva convention, You, otoh, might or might not be a bot. We’re still checking, That tiny buzz you hear in your hard drive is us. Take no notice. We mean you no harm.
Rich,
Aboslutely correct — I was thinking “direction and timeframe” and my fingers failed to comply.
Since no one can model a accurate economic scenarios, planned economies have famines.
@49erDweet:
I’ve failed the turing test… this is the ultimate humiliation for a troll-bot.
@Doug M
I don’t want to believe that accurate modelling of an economy is impossible. I’m not suggesting a model that will work skillfully for the next 10 years, but perhaps one that has a shelf life of six months is plausible.
There is a model [training/fittting/building/learning] method called ‘online learning’. The idea is that the model is updated as time passes. In these sorts of systems one abandons the notion that the model will explain anything about the dynamics of a system. The only concern is that the model skillfully predicts future outcomes.
Will,
Your “online learning” model sounds it just gets lucky now and then when tomorrow looks like today.
OBEliza: Does that trouble you?
Matt: I cant take credit for the idea; it’s been around at least 20 years longer than I have. 🙂
Seriously though: Why does it seem so outlandish to simply never stop presenting training samples? Most neural networks are built using an online learning algorithm (back propagation. Works one sample at a time and the Learning rate is a function of when, not what.).
Online learning is used by many social network and forum focused data mining solutions (since language, slang, persona, and topics are constantly evolving). They are useful when the dynamics of a particular problem are changing, but not so quickly that the data of interest is entirely random.
> Parity error. 01000111010100010
I’m quite curious to see where you are going with this, Dr. B., but I think that “vague prediction” (Matt’s definition) is what most people would call the thing you are referring to as a “scenario”.
I’m wondering if you’re not creating some confusion by conflating “scenario” and “prediction” as the words are used in practice by most scientists. At least with respect to modeling, scenarios are often (mainly?) used to assess model sensitivity or to constrain some quantity of interest, such as atmospheric CO2 concentration. Perhaps where you are headed is to say that we should only refer to the model output itself a “scenario”, and never a “prediction, though many people would probably be satisfied equating “prediction” with “most likely output scenario”. Let’s see what Part IV says.
Will,
Yes, I’m familiar with the concept.
Firstly, I would say that neural networks are notoriously fragile. Even minute deviations from the training data cause them to fail.
Data mining techniques are good for looking for patterns in the existing data, and that’s a useful thing to do, but it’s not the same thing as creating something with skill at predicting the future. I need my spam filter to be good at finding today’s spam, but I don’t expect it to be able to predict what some guy in Nigeria will think of next.
@Matt: I don’t see the difference between a model predicting a response based on X[i], or a response based on X[i-1]. All models are making predictions.
Briggs: “Scenarios are predictions lite. They have conditions, just like predictions do, but one is not meant to look too closely at them.”
Hmmm. . .
As I said in response to the prior post, the existence of conditions is not the important fact. I understand you agree. What you seem to be saying then, is that the amount of vagueness in defining either the conditions or the result is what takes a statement from the realm of prediction and makes it a scenario.
I suppose we could take the word “scenario” and redefine it to mean “vague prediction,” which would then support your view of the word, but I don’t think there is any need to redefine a perfectly good word in such a way that it loses its individual meaning and just becomes a “vague” adjective attached to “prediction.”
Indeed, a scenario could certainly be very precise and detailed. We could lay out the parameters with incredible precision, like we do in weapons design, spacecraft flights, etc. We can get very detailed and specific — not vague at all — and lay out scenarios, things that are predicted to occur if certain, very specific and well-defined conditions attain.
The key distinction is not in the existence or absence of conditions. Nor is it in the vagueness or specificity of the definitions.
There’s nothing wrong with the predictions in Chinese fortune cookies. It’s just that you get the wrong cookie. 🙂