Skip to content
February 10, 2009 | 31 Comments

Theory confirmation and disconfirmation

Time for an incomplete mini-lesson on theory confirmation and disconfirmation.

Suppose you have a theory, or model, about how some thing works. That thing might be global warming, stock market prices, stimulating economic activity, psychic mind reading, and on and on.

There will be available a set of historical data and facts that lead to the creation of your theory. You will always find it an easy process to look at those historical data and say to yourself, “My, those data back my theory up pretty well. I am surely right about what drives stock prices, etc. I am happy.”

Call, for ease, your theory MY_THEORY.

It is usually true that if the thing you are interested in is complicated—like the global climate system or the stock market—somebody else will have a rival theory. There may be several rival theories, but let’s look at only one. Call it RIVAL_THEORY.

The creator of RIVAL_THEORY will say to himself, “My, the historical data back my theory up pretty well, too. I am surely right about what drives stock prices, etc. I am happy and the other theory is surely wrong.”

We have a dispute. Both you and your rival are claiming correctness; however, you cannot both be right. At least one, and possibly both, of you is wrong.

As long as we are talking about historical data, experience and human nature shows that the dispute is rarely allayed. What happens, of course, is that the gap between the two theories actually widens, at least in the sense of strength which the theories are believed by the two sides.

This is because it is easy to manipulate, dismiss as irrelevant, recast, or interpret historical data so that it fits what your theory predicts. The more complex the thing of interest, the easier it is to do this, and so the more confidence people have in their theory. There is obviously much more that can be said about this, but common experience shows this is true.

What we need is a way to distinguish the accuracy of the two theories. Because the historical data won’t do, we need to look to data not yet seen, which is usually future data. That is, we need to ask for forecasts or predictions.

Here are some truths about forecasts and theories:

If MY_THEORY says X will happen and X does not happen, then MY_THEORY is wrong. It is false. MY_THEORY should be abandoned, forgotten, dismissed, disparaged, disputed, dumped. We can say that MY_THEORY has been falsified.

For example, if MY_THEORY is about global warming and it predicted X = “The global mean temperature in 2008 will be higher than in 2007” then MY_THEORY is wrong and should be abandoned.

You might say that, “Yes, MY_THEORY said X would happen and it did not. But I do not have to abandon MY_THEORY. I will just adapt it.”

This can be fine, but the adapted theory is no longer MY_THEORY. MY_THEORY is MY_THEORY. The adapted, or changed, or modified theory is different. It is NEW_THEORY and it is not MY_THEORY, no matter how slight the adaptation. And NEW_THEORY has not made any new predictions. It has merely explained historical data (X is now historical data).

It might be that RIVAL_THEORY theory made the same prediction about X. Then both theories are wrong. But people have a defense mechanism that they invoke in such cases. They say to themselves, “I cannot think of any other theory besides MY_THEORY and RIVAL_THEORY, therefore one of these must be correct. I will therefore still believe MY_THEORY.”

This is the What Else Could It Be? mechanism and it is pernicious. I should not have to point out that because you, intelligent as you are, cannot think of an alternate explanation for X does not mean that an alternate explanation does not exist.

It might be that MY_THEORY predicted Y and Y happened. The good news is that we are now more confident that MY_THEORY is correct. But suppose it turned out that RIVAL_THEORY also predicted that Y would happen. The bad news is that you are now more confident that RIVAL_THEORY is correct, too. How can that be when the two theories are different?

It is a sad and inescapable fact that for any set of data, historical and future, there can exist an infinite number of theories that equally well explain and predict it. Unfortunately, just because MY_THEORY made a correct prediction does not imply that MY_THEORY is certainly correct: it just means that it is not certainly wrong. We must look outside this data to the constructs of our theory to say why we prefer MY_THEORY above the others. Obviously, much more can be said about this.

It is often the case that a love affair develops between MY_THEORY and its creator. Love is truly blind. The creator will not accept any evidence against MY_THEORY. He will allow the forecast for X, but when X does not happen, he will say it was not that X did not happen, but the X I predicted was different. He will say that, if you look closely, MY_THEORY actually predicted X would not happen. Since this is usually too patently false, he will probably alter tactics and say instead that it was not a fair forecast as he did not say “time in”, or this or that changed during the time we were waiting for X, or X was measured incorrectly, or something intervened and made X miss its mark, or any of a number of things. The power of invention here is stronger than you might imagine. Creators will do anything but admit what is obvious because of the passion and the belief that MY_THEORY must be true.

Some theories are more subtle and do not speak in absolutes. For example, MY_THEORY might say “There is a 90% chance that X will happen.” When X does not happen, is MY_THEORY wrong?

Notice that MY_THEORY was careful to say that X might not happen. So is MY_THEORY correct? It is neither right or wrong at this point.

It turns out that it is impossible to falsify theories that make predictions that are probabilistic. But it also that case that, for most things, theories that make probabilistic predictions are better than those that do not (those that just say events like X certainly will or certainly will not happen).

If it already wasn’t, it begins to get complicated at this point. In order to say anything about the correctness of MY_THEORY, we now need to have several forecasts in hand. Each of these forecasts will have a probability (that “90% chance”) attached, and we will have to use special methods to match these probabilities with the actual outcomes.

It might be the case that MY_THEORY is never that close in the sense that its forecasts were never quite right, but it might still be useful to somebody who needs to make decisions about the thing MY_THEORY predicts. To measure usefulness is even more complicated than measuring accuracy. If MY_THEORY is accurate more often or useful more often, then we have more confidence that MY_THEORY is true, without ever knowing with certainty that MY_THEORY is true.

The best thing we can do is to compare MY_THEORY to other theories, like RIVAL_THEORY, or to other theories that are very simpler in structure but are natural rivals. As mentioned above, this is because we have to remember that many theories might make the same predictions, so that we have to look outside that theory to see how it fits in with what else we know. Simpler theories that make just as accurate predictions as complicated theories more often turn out to be correct (but not, obviously, always).

For example, if MY_THEORY is a theory of global warming that says that there is a 80% chance that global average temperatures will increase each year, we need to find a simple, natural rival to this theory so that we can compare MY_THEORY against it. The SIMPLE_THEORY might state “there is a 50% chance that global average temperatures will increase each year.” Or it might be that LAST_YEAR’S_THEORY might states “this year’s temperatures will look like last year’s.”

Thus, especially in complex situations, we should always ask, when somebody is touting a theory, how well does that theory make predictions and how much better is it than its simpler, natural rivals. If the creator of the touted theory cannot answer these questions, you are wise to be suspicious of that the theory and to wait until that evidence comes in.

February 9, 2009 | 1 Comment

Panda Runner

The site Panda Runner went live over the weekend.

You haven’t been there yet? Now’s your chance.

The site is a collaboration between my number one son and his best friend. The back end of the site was mostly built by my number two son.

It just started so they don’t have a ton of stuff yet, but they plan to.

Here’s what they say about themselves:

Why is PandaRunner the best place to buy Asian products?

1. We’ve got unique products, some of which can only be purchased in North America from our online shop.

2. We care about price and quality just like you. Our team spends a lot of time tracking down quality products that don’t cost a fortune.

3. Our site makes it easy to pick up hot Asian goods with our innovative shopping cart and checkout screen.

Stop by if you get a chance, or drop them a line about what kind of things you’d like to see.

Thanks everybody!

February 7, 2009 | 11 Comments

Some good economic news!

The employment rate is plunging faster than a hundred kilogram bolide entering the atmosphere at a normal angle to the plane surface of the Earth [thank 49er!].

Manufacturing is down over a million, with Construction close behind. Plain Business and and Retail are close behind, each losing hundreds of thousands. Even the Hospitality sector lost over a quarter million slots.

But, lo!, do not despair! For there is at least one area that has seen an increase in the number of jobs.

That area? Government.

According to the Labor Department, the Drain To Our Wallets Sector increased the number of luke-warm bodies it employs by about 170,000 since December.

Problem with that number, and all the numbers the Labor Department issues, is that they are “seasonally adjusted.” Which means that the numbers aren’t real numbers, they are outputs from some statistical model. We don’t know what the actual numbers are.

What happens is that unemployment usually decreases in the first part of the year, due to things like retail letting go of the temporary workers it hired for the Federally-Recognized Holiday of December 25th rush, etc. The opposite is also true: employment grows in the weeks leading up to December.

The Labor Department, and economists elsewhere, don’t like to see these dips and doodles and so they apply a statistical filter that brings the unemployment rate up when it usually goes down, and down in when it usually goes up.

These are weird statistical models. They take as input the real numbers and spit out not-real numbers. Economists worry people won’t understand the natural up- and down-swings in unemployment and so massage the real numbers to make them less variable, and, presumably, more calming.

Anyway, the real number of bodies tossed onto the street is different than the numbers you read in the press. I don’t know by how much. Just something to keep in mind.

Point is, it’s not all doom and gloom out there. The government is growing, and that must be a good thing, right?

February 5, 2009 | 17 Comments

I should be happier

According to and the Wall Street Journal, I should be one happy guy. This is because Career Cast has done a “study” to rate the best jobs, and mine is right up there.

Number 1? Mathematician. Which is partly me. Number 2 is actuary, which is barely different than number 3, statistician, which is me all over. But we also have 12 (philosopher), 13 (physicist), and 15 (meteorologist), which are various flavors of me.

Six out of the top fifteen ought to see me skipping down the street each day (if I didn’t worry about slipping on the ice), smelling the flowers (if there were any in this deep freeze), smiling at children (who dangerously crowd the sidewalk), and whistling happy tunes (I worry my lips would freeze together).

Yet I am less than sanguine.

Maybe it is because I don’t completely trust the “study.” After all, job number 14 is “parole officer”, which, if Career Cast is right, is better than being a weatherman. Sitting on your wallet hoping your “client” doesn’t find out where you live is better than explaining vorticity to somebody?

How did they come up with these numbers anyway? Well, being in the top three is supposed to guarantee a job “indoors and in places free of toxic fumes or noise.” Obviously, these people have never been to a hospital before, where I work. The scents emanating from the public toilet outside our ED are so obnoxious they are actually visible.

The last job in the list, number 200, is Lumberjack. What boy, or man remembering being a boy, doesn’t want to be a lumberjack? Career Cast cares about safety. They worry you might stub your toe tripping over a log. I’d care about dropping enormous trees with pinpoint precision with just a chainsaw and then heading to the bar for a cold one. Or two.

Number 199 is Dairy Farmer. True, you’re going to see a lot of cow shit, but it’s beats the stuff they continuously shovel at job number 10, Accountant.

Some others in the bottom twenty: Seaman, Roofer, Welder, Auto Mechanic, Butcher, Fireman, Garbage Collector.

For the last eleven years I have seen the same two guys drive their garbage truck down my street. They are always chatting and appear happy. They are outside and not hunched in front of a computer. There’s very little stress. They get good pay and benefits and first dibs at any choice garbage1. They don’t need to shell out cash for a gym membership to “exercise”, which is better defined as work that you pay for. In every parade I’ve ever been to, it’s the garbage men following after the horses that get the biggest applause.

Butcher? Free meat and access to sausage of every type, being covered with blood in a manly way, and flirting with housewives. Auto Mechanic? The satisfaction of fixing something with your hands, the chance of custom jobs, the relish of being needed. Fireman? The benefits are too plain to mention.

All the jobs at the bottom are those that scored low on Career Cast’s “Physical demands (crawling, stooping bending, etc.)” index. Somehow, these keyboard punchers decided that these were negatives instead of the positives they truly are. It’s easy to imagine the folks over there have never gotten their hands dirty and view the prospect with horror.

For me, I’d rather be a Construction Worker like my dad (number 190) than a Medical Lab Technician (number 16) any day.


1I think I stole this line from Mike Royko.