In Part I of this post, we started with a typical problem: which of two advertising campaigns was “better” in terms of generating more sales. Campaigns A and B were each tested for 20 days, during which time sales data was collected. The mean sales during Campaign A was $421 and the mean sales during Campaign B was $440.

Campaign B looks better on this evidence, doesn’t it? But suppose instead of 20 days, we only ran the campaigns one day each, and that the sales for A was just $421 and that for B was $440. B is still better, but our intuition tells us that the evidence isn’t as strong because the difference might be due to something other than differences in the ad campaigns themselves. One day’s worth of data just isn’t enough to convince us that B is truly better. But is 20 days enough?

Maybe. How can we tell? This is the part that Statistics plays. And it turns out that this is no easy problem. But please stay with me, because failing to understand how to properly answer this question leads to the most common mistake made in statistics. If you routinely use statistical models to make decisions like this—”Which campaign should I go with?”, “Which drug is better?”, “Which product do customers really prefer?”—you’re probably making this mistake too.

In Part I, we started by assuming that the (observable) sales data could be described by probability models. A probability model gives the chance that the data can take any value. For example, we could calculate the probability that the sales in Campaign A was greater than $500. We usually write this using math symbols like this:

Pr(Sales in Campaign A > $500 | e)

Most of that formula should make sense to you, except for the right-hand side of it. The bar at the end, the “|”, is the “given” bar. It means that whatever appears to the right of it is accepted as true. The “e” is whatever evidence we might have, or think is true. We can ignore that part for the moment, because what we really want to know is

Pr(Sales in B > Sales in A | data collected)

But that turns out to be a question that is impossible to answer using classical statistics!

Continue Reading »

Here’s a common, classical statistics problem. Uncle Ted’s chain of Kill ‘em and Grill ‘em Venison Burgers tested two ad campaigns, A and B, and measured the sales of sausage sandwiches for 20 days under both campaigns. This was done, and it was found that mean(A) = 421, and mean(B) = 440. The question is: are the campaigns different?

In Part II of this post, I will ask the following, which is not a trick question: what is the probability that mean(A) < mean(B)? The answer will surprise you.

But for right now, I merely want to characterize the sales of sausages under Campaigns A and B. Rule #1 is always look at your data! So we start with some simple plots:

I will explain box and density plots elsewhere; but for short: these pictures show the range and variability of the actual observed sales for the 20 days of the ad campaigns. Both plots show the range and frequency of the sales, but show it in different ways. Even if you don’t understand these plots well, you can see that the sales under the two campaigns was different. Let’s concentrate on Campaign A.

This is where it starts to get hard, because we first need to understand that, in statistics, data is described by probability distributions, which are mathematical formulas that characterize pictures like those above. The most common probability distribution is the normal, the familiar bell-shaped curve.

The classical way to begin is to then assume that the sales, in A (and B too), follow a normal distribution. The plots give us some evidence that this assumption is not terrible—the data is sort of bell-shaped—but not perfectly so. But this slight deviation from the assumptions is not the problem, yet.

Continue Reading »

My paper on this subject will finally appear in the Journal of Climate soon. You can see it’s status (temporarily, anyway) at this link.

You can download the paper here.

The gist is that the evidence shows that hurricanes have not increased in either number of intensity in the North Atlantic. I’ve only used data through 2006; which is to say, not this year’s. But if I were to, then, since the number and intensity of storms this past year were nothing special, the evidence would be even more conclusive that not much is going on.

Now, I did find that there were some changes in certain characteristics of North Atlantic storms. There is some evidence that the probability that strong (what are called Category 4 or 5) storms evolving from ordinary hurricanes has increased. But, there has also been an increase in storms not reaching hurricane level. Which is to say, that the only clear signal is that there has been an increase in the variability of intensity of tropical cyclones.

Of course, I do not say why this increase has happened. Well, I suggest why it has: changes in instrumentation quality and frequency since the late 1960s (which is when satellites first went up, allowing us to finally observe better). This is in line with what others, like Chris Landsea at the Hurricane Center, have found.

I also have done the same set of models of global hurricanes. I found the same thing. I’m scheduled to give a talk on this at the American Meteorological Society’s annual meeting in January 2008 in New Orleans. That paper is here.

This is a lovely, lovely book and I can’t believe it has taken me this long to find and read it (November 2005: I was lead to this book via Jaynes, who was the author that also recommended Stove). Cox, a physicist, builds the foundations of logical probability using Boolean algebra and just two axioms, which are so concise and intuitive that I repeat them here:

1. “The probability of an inference on given evidence determines the probability of its contradictory on the same evidence.”

2. “The probability on given evidence that both of two inferences are true is determined by their separate probabilities, one on the given evidence, the other on this evidence with the additional assumption that the first inference is true.”

Cox then begins to build. He shows that probability can be, should be, and is represented by logic; he shows the type of function probability is, the relation of uncertainty and entropy, and what expectation is. He ends with deriving Lapace’s rule of succession, and argues when this rule is valid and when it is invalid. And he does it all in only 96 pages!. This is one of the rare books that I also recommend you read each footnote. If you have any interest in probability or statistics, you have a moral obligation to read this book.

How does Santa Claus do it? How does he get all those presents to all those kids in just one night? Some people think that the old man still personally delivers all those presents just by himself. But there are too many kids now, so the traditional method has become impossible. So I was asked by the show Weird US to outline the modern mathematical ideas that Santa Claus actually uses.

There is no way Santa could physically deliver all the presents in just one night. This is because there are tens to hundreds of millions of children, and there is not enough time, energy, or space to complete the task in this short a time. A typical mathemtical analysis is this one by an engineer. His math and reasoning are flawless: any argument based on speed fails, which of course it must. However, those presents do get there, so Santa must be doing something. But what?

Have you see the movie Miracle on 34th Street? I mean the original, not any of the (unnecessary) remakes. There is a scene in the sanity trial of the old man who claims to be Santa in which the defense attorney calls to the stand the young son of the prosecutor. The prosecutor has previously argued that there is no Santa Claus. The defense attorney, John Payne, asks, (words to the effect), “Johnny, do you believe in Santa Claus?” The kid replies, “Sure I do.” Payne: “Why?” Kid: “Because my daddy told me (there was a Santa Claus).” Payne: “And your daddy is a very honest man, isn’t he? He wouldn’t lie?” Kid: “My daddy would never lie, would you daddy?” The kid comes off the stand and whispers to Santa that he’d like a football for Christmas.

Well, we all know what happens. The prosecutor conceeds the existence of Santa and the court eventually decides that the old man in the dock is the one and only Santa Claus. But the key scene sneaks by unless you’re paying close attention. It’s when the case is over and people are noisly exiting the courtroom. We see the prosecutor suddenly realize that he’s got to run. He says to himself, “I’ve got to hurry if I’m going to get that football helmut!”

To be obvious: the kids asks Santa for the helmut, but it is the father who brings it. Do you see? Santa manipulated the events so that the kid got what he wanted for Christmas — Santa was responsible for the present — but Santa did not actually, physically have to bring the present. Here’s how.

Continue Reading »

In the Tuesday, 6 November 2007 edition of the Wall Street Journal, Pfizer took out a full-page ad encouraging people to “Ask your doctor” about Lipitor, a drug which claims to lower your “risk” of a heart attack (p. A13). In enormous bold print are the words:

Lipitor reduces risk of heart attack by 36%*.

.

Following that asterisk on the 36% leads to something interesting:

*That means in a large clinical study, 3% of patients taking a sugar pill or placebo had a heart attack compared to 2% of patients taking Lipitor.

.

Congratulations, Pfizer! This ad scores a solid 7 on the Statistical Deception Scale.

First, if you take Lipitor your risk is only lowered by a relative amount, from an already low 2% to a slightly lower 1%. Your real risk is only lowered by 1%. There is a world of difference between that 36% and 1%, and the ad did say, sort of, that the risk was relatively lowered, not lowered absolutely, so it wasn’t terribly deceptive at that point. It’s true, too, that some people might think to themselves, “Ah, any lowering is good, even if it’s only 1%.” More on that sentiment in a moment.

But what most people won’t see, or will ignore, are the smaller words under the bold headline, which say that your risk is lowered “If you have risk factors such as family history, high blood pressure, age, low HDL (’good’ cholesterol) or smoking.” Aha! This is what ups Pfizer’s ranking on the deception scale.

Thus, in order to get the 1% reduction it turns out that you have to be in a pretty high risk group to begin with; namely, those with “multiple” risk factors. How many risk factors do you have to have before you can hope for the reduction? Two? Three? The ad doesn’t say. Maybe you need all five before you can hope for the reduction. That is the most likely reading of the ad.

What if you don’t have all five? We might guess that your absolute risk reduction is either zero or negligible. We guess this because if people could reduce their risk generally, without belonging to a highly selective group, that Pfizer would have boasted of this. They did not so boast, so etc. etc.

Back to the “any lowering is good” sentiment. On the page opposite the pictures, Pfizer has quite a long list called “POSSIBLE SIDE EFFECTS OF LIPITOR”. Among these new risks are, muscle problems, kidney “problems” or even failure, liver problems, nausea, vomiting, brown colored urine, tiredness, yellowing eyes (!), rash, gas, and others. The key words are these:

Fewer than 3 people out of 100 stopped taking LIPITOR because of side effects.

Well, must I point out that 3 out of 100 is 3%, which is more—67% more!—than the 2% (in the high risk group) who will have a heart attack, and 200% more than the 1% (or so) of the “regular” people who will have a heart attack? I guess I don’t need to. Of course, we can’t figure out, given only the data that Pfizer provides in the ad, what the actual chance is that a regular person will have a heart attack or suffer “side” effects. But there is enough information provided that should severely limit enthusiasm for this drug.

The advertisement also pictures Robert Jarvik, inventor of the “Jarvik artificial heart”, badly in need of a haircut, standing in front of a colorful heart-like object. But he is a celebrity doctor, and isn’t that all you really need to know?