Not all models are false. So much we have discussed before.
To say a model is false is to show that one of the premises which comprise the model is false. There are thus certainly many false models. But then there are many true ones. Casinos positively rely on them—and make, as my sister would say, beaucoup bucks doing so. It would be too tedious to rehearse the premises of dice throws and the like. Regular readers will know them off their hearts. But for the freshly minted, this series is pretty good showing some true models.
In order to say a thing is false we must have proof in the form of a logically valid argument. It is no good whatsoever to say, “Ah, the model can’t be true.” This isn’t a valid argument, even though, stripped down, it is the most popular one.
One such proof would be in the form, “The third premise in the model is false because of this true condition.” If such proof isn’t forthcoming, we don’t know the model is false. And some models we can even prove are true, as just said. These start with accepted premises and move to a conclusion the probability of which can be deduced. This probability can even be 1, as in mathematical theorems.
Mathematicians, even of the statistical variety, are comfortable with proof and use it often. But not when it comes to saying this or that model is false.
A true model is not falsified when it says, “The probability of C is X” where C is some proposition and X is some number between 0 and 1, because no observation of C or not-C can ever falsify that model. If a model says “The probability of C is 0” or 1, or these numbers are in its boundaries, then that model can be falsified if it says C is impossible but C obtains. But it cannot always be falsified if it says C is certain and we don’t see C—unless contained in C are other indicators, such as timeliness.
A model cannot be falsified if it says “The probability of C is exceedingly small” and then C is later found to be be true, because the model did not say C was impossible. It is here where many mistakes are made.
Now many, many models statisticians use are false. Every “normal” model is, which includes all regressions and so forth, and these comprise the bulk of workaday models. Every normal model eventually says, “The probability of C is 0” where C is almost any proposition, and where C’s “pop up” with regularity. For example, in modeling temperature a normal is often used, and this model must say, “The probability the temperature will be T is 0” for any T. Yet we will certainly see some T, and when we do we have immediately falsified the model.
Since normal and other “continuous” models, which are models incorporating mathematical infinities and used on finite realities, are used, they are always false. This may be where the perception that “all” models are false originated.
The second portions of Box’s quip is that false models can be useful. Here is where it gets interesting. Take any normal model which is used in practice. If any thought about its validity is made it will be realized the model is false. But it will still be used. Mostly because of inertia or custom, but also because of the sense that the model is “close enough.”
It is that unquantified “close enough” which is fascinating. The average user of statistical models never considers it, at least not seriously. And the statistician is usually satisfied if his math works out or that his simulations are pretty (not worrying that many of these run dangerously close to, or actually are, instances of circular logic).
It is true that sometimes “close enough” is indeed close enough. But since many models don’t get real checking on truly independent evidence, many times “close” isn’t even in the general vicinity.
This is why predictive inference is such a good idea. I often give examples using regression which show both classical frequentist and Bayesian results are “good” in the sense indicated in those theories. But when the model is used in its real-life sense—i.e. its predictive mode, making probability statements about the same observables that were modeled, which after all is the reason for the model in the first place—then it becomes glaringly obvious the models stink worse than a skunk on the side of the road in August.
Examples to come!
Well said!
I am reminded of William of Ockham’s comments on models: They should not have more terms (entities) than necessary because you will not then understand your own model. The real world, he went on to say, might be as complex as God wills. George Box was one of the few scientists in the Modern Ages who actually understood Ockham’s Razor.
“All models are wrong,” therefore, in the sense that all models are approximations and cannot, in the name of intelligibility, contain as many factors as the Thing being modeled. In an assessment of the nitrogen content of heats of a certain kind of steel, a histogram revealed a normalish-looking picture, and a normal approximation indeed provided decent predictive inferences concerning the proportion of heats failing to meet the nitrogen content specification. But the fish stinks from the head down, and models begin to stink at their extreme values. The farther out one goes, the “wronger” the predictions become. The normal distribution — any distribution — goes off to infinity in both directions, and no electric arc furnace ever built is likely to match that feat. (Of feet, since we are talking about extremities.)
This is a different perspective than the technical objection that P(x)=0 at any given dimensionless point x. Those dimensionless points are themselves models invented by Euclid. Whitehead noted in his theory of relativity that in the real world we always deal with Events, not with Points, and events always have duration. We were not interested in P(x=90ppm) but in P(x>90ppm).
Difficulties ensue when we mistake the outputs of a model for the behavior of the world.
The way I learned it you never (as in never) use the normal distribution to derive the probability of some point value but always to determine the probability that the value falls within a given range. And that this is true for any continuous distribution. Aren’t you disproving something nobody says?
YOS,
Right. Many normal and other continuous models do okay at intervals, as long as the intervals are sufficiently wide. Any greater-than or less-than interval, since infinity is on the other side, is usually okay. But narrow intervals aren’t so good. Physical processes are also better behaved than sociological ones. Anything to do with people is dicey at best.
And then we have the problem of observation. Continuous models says nothing can be observed, yet we must observe to model. This is okay when the things of interest are “more or less” continuous, like (say) movement of some tiny thing through some controlled force. But it’s bad to worse for really discrete things, like answers on questionnaires (“On a scale of 1 to 4…”).
But in any case, looking at what the model actually infers about the real observable—the real thing of interest—is always better than looking at parameters or statistics.