William M. Briggs

Statistician to the Stars!

Page 148 of 581

Use The Wrong P-value, Go To Jail: Not A Joke: Updated With Amicus Brief

This is what over-exposure to p-values can lead to.

This is what over-exposure to p-values can lead to.

Today’s lesson: If the government wants you bad enough, it will get you. If that isn’t already obvious, consider what befell W. Scott “Don’t Call Me Baron” Harkonen.

Just kidding with the Dune reference. Harkonen was imprisoned by the Padishah—stop that!—by our beneficent government for the most heinous crime of using a p-value which his competitors did not like.

I do not joke nor jest. Harkonen got six months house arrest for writing these words in a press release:

InterMune Announces Phase III Data Demonstrating Survival Benefit of Actimmune in IPF [idiopathic pulmonary fibrosis]. Reduces Mortality by 70% in Patients with Mild to Moderate Disease.

According to the Washington Post,”What’s unusual is that everyone agrees there weren’t any factual errors in the [press release]. The numbers were right; it’s the interpretation of them that was deemed criminal.” Post further said, “There was some talk that if Harkonen had just admitted more uncertainty in the press release—using the verb ‘suggest’ rather than ‘demonstrate’—he might have avoided prosecution.”

Harkonen followed FDA-government rules and ran a trial of his company’s drug actimmune (interferon gamma-1b) in treating IPF, hoping patients who got the drug would live longer than those fed a placebo. This happened: 46% of actimmune patients kicked over while 52% of the placebo patients handed in their dinner pails.

Unfortunately, the p-value for this observed difference was just slightly higher than the magic number: it was 0.08.

Wait! Tell me the practical difference between 0.08 and the magic number? You cannot do so. That is what makes the magic number magic. Occult thinking is rife in classical statistics. There is no justification given for the magic of the magic number other than it is magic. And it is magic because other people, Bene Gesserit fashion (last one), have said it is magic.

Therefore, p-values greater than the magic number are “insignificant.” The FDA shuns p-values that don’t fit into the special magic slot. Harkonen, holding his extra-large p-value, knew this. And wept.

I’m guessing about the weeping. But Harkonen surely knew about the mystical threshold, because he dove back into his data where he discovered that the survival difference in patients with “mild to moderate cases of the disease” was even greater, a difference which gave the splendiferously magical p-value of 0.004.

So wee was this new p-value and so giddy was Harkonen that he wrote that press release.

Which caught the attention of his enemies (rival drug company?) who ratted him out to the Justice Department’s office of consumer litigation, which, being populated by lawyers paid to snare citizens, did their duty on Harkonen.

Harkonen’s crime? Well, in classical statistics the pre-announced “primary endpoint”, what happened to all and not a subset of patients, is the only thing that should have counted. The “secondary analysis”, especially when it’s not expected, is feared and should not be used.

And rightly so when using p-values, because as long as the data set is large and rich enough, wee p-values can always be discovered even when nothing is happening, which in this case means even when the drug doesn’t work. The government therefore assumed the drug didn’t work and that Harkonen should not have used the word “demonstrated”, which it interpreted as meaning “a wee p-value less than the magic number was found.”

What makes the story pathetic is that Harkonen forgot when he got his 0.08 that the p-value is dependent on the model he picked. He could have picked another, one which gave him a smaller p-value. He could have kept searching for models until one issued a magic p-value. He might not have found one, but there’s so many different classical test statistics that it would have been worth looking.

Which of these p-values is “the” correct one? All of them!

Insult onto injury time. As Harkonen rattled his coffee cup against his mullions (house arrest, remember), his old company did a new, bigger trial on just the subset of patients who did better before. Result: more deaths in the drug than placebo group. Oops.

Anyway, maybe we should let the government, for a limited period of time, arrest and jail scientists who publicly boast of wee p-values and whose theories turn out to be garbage. Nah. Our prisons aren’t nearly big enough to handle it.

Return to this page Sunday for the highly anticipated post: Everything Wrong with P-Values Under One Roof.

Update Don’t miss the comment by Nathan Schachtman, who filed an amicus brief on Harkonen’s behalf. It’s linked below.


Thanks to Al Perrella for finding this.


On The Evidence From Experiments: Part III

Typical non-cancerous albondigas.

Typical non-cancerous albondigas.

Read Part I, Part II first.

All other propositions are contingent. For example, it is not necessarily true that your treatment should be a cure for cancer of the albondigas yet it might be contingently true that it does. It is not necessarily true because there is no chain of argument, as there is for (say) a mathematical proof, that lands on indubitable axioms which taken as a whole proves that it cannot be other than that your treatment cures.

Here is an example from Lewis Carroll. It is contingently true that C = “Some chickens are creatures understanding French” if we accept as true for the sake of argument the (compound) proposition “All cats are creatures understanding French and some chickens are cats.” Our C is not, however, necessarily true because the premises which we assumed true are not true in fact. That means the chain of argument eventually leads to a falsity (no chickens are cats).

Leading to an uncertainty would amount to the same thing, in the sense that we would know we are dealing with a contingent proposition. Thus “working”, scientific, and observational propositions are uncertain and contingent.

A corollary to this is that a contingent truth is easily transformed into a contingent falsity or into a proposition which is uncertain. Keep C above and change the premises/evidence to “No cats are creatures understanding French and all chickens are cats” then it is contingently false that “Some chickens are creatures understanding French.” (Homework: why the two modifications?)

It should be obvious how to make any contingent proposition true, false, or in between by a simple choice of evidence. Since the evidence of contingent statements is rarely universally agreed upon, it should come as no surprise that disagreements can always exist about the truth or uncertainty of contingent propositions.

The real difficulty is in failing to monitor our language and the resultant over-confidence. We often speak loosely of contingent propositions, that they are “true” or “false” or “everybody knows that…” These are always strictly mistakes; but in everyday speech the consequences are trivial or insignificant. These are the “true enoughs” or propositions which have a chain of evidence sufficiently strong that they may as well be necessarily true, but aren’t quite: “I’m reading these words on an electronic device” (you may be deceived), “We’re having hamburgers for dinner,” “My car is in the driveway,” “Mars is the fourth planet from the sun,” and so forth almost endlessly.

But there are other examples much farther from true, most of which occur in politics, the edges of science, popular morality, etc. Example: “It is beyond doubt that ‘Mankind is causing harmful climate change.'” The speaker of this evidently has premises in mind which, if accepted, leads to the contingent truth “Mankind is causing harmful climate change.” What the speaker doesn’t realize is that his evidence does not have to be accepted, indeed unequivocally cannot be, for his opponent can supply different evidence which makes the proposition close to false.

Yet both sides argue about the probability the proposition is true—“95%!” “10%!”—forgetting the real battle is over the evidence. Because once the evidence is settled and agreed upon by all, the probability the proposition is true follows deductively (it may not, of course, be a number).

Short recap, since this subject is not easy. Any proposition depends for its degree of truth on specified evidence, or premises. Anybody is free to supply or change these (for a fixed proposition of interest). Even “1 + 1″ does not have to equal 2 conditioned on premises other than the normal ones. Yet for some propositions (like “1+1=2″), called necessary truths, there is a chain of evidence which is obviously correct and not “substitutable” and which, taken as a whole, shows that the proposition must be true, that it cannot be false or in-between.

Contingent propositions, which are most in life, do not have a chain of evidence which proves the proposition of interest is necessarily true or false. Contingent propositions may, like necessary truths, have a chain of evidence on which everybody agrees but which only show the proposition has some non-extreme probability (which may be a number or interval or no number at all). Sometimes truth is denied us. This is Tough Luck.

On the other hand, the largest class of contingent propositions have no premises on which all agree. Hence disputes, acrimony, indigestion. But there are usually clues. Evidence accepted for one proposition may also figure as evidence for a second and third proposition, which themselves have significant support. This makes it more likely the evidence for the first proposition will be accepted, as long as the propositions taken together are said to be in a “class.” Susan Haack thus likens our knowledge to an enormous crossword puzzle, where entries have to make sense in more than one direction—but where some of the clues are missing! More on this another day.

So what does all this have to do with the Bobs and cancer and of the albondigas? Everything.


The Case of the Missing Global Warming: A 17th Precinct Mini Mystery

A 17th Precinct Mini Mystery

This originally ran four years ago on 21 September 2009. One or two minor details have been changed to make the post current. Update: I see I screwed up the update and lost the copy where it was Michael Mann calling in the report. I have no luck with WordPress’s scheduler. Use your imagination.

“Hey, Sarge. Got a lady here who wants to file a missing persons report…Sarge?” Officer Hannigan stood in front of Sergeant Fitzgerald’s desk and rustled a sheaf of paper just loud enough so that it didn’t sound intentional, but with enough force to be heard.

Sergeant Fitzgerald was dozing and he almost started at the noise, but long experience enabled him to remain still. He did not want his junior to know he had been asleep, so he counted to three then slowly made the sign of the cross and said, “Amen.” He let his watery eyes find Hannigan’s.

“Oh, sorry, Sarge.” Hannigan was new enough not to have seen the act before. “But I got this strange call and I didn’t know what to do.” Fitzgerald raised both eyebrows a millimeter. “This lady wants to report a missing person, only…”

Enough consciousness had seeped into Fitzgerald’s being that he was able to slap the table. “Now, young Hannigan. Nothing could be easier. You have the right forms?” A nod. “You’ve followed procedure and asked the right questions?”

“I have.”

“Then there is no problem.” He shifted his weight and turned his attention inward.

“But Sarge, the answers made no sense!”

Fitzgerald sighed and knew that sleep was banished. “Well, then. Let’s have it. Who’s missing?”

“Global Warming.”

“And what’s that, then?” A shrug was his answer. He sighed. “How long has it been missing?”

“Lady said about sixteen years, maybe seventeen.”

“Seventeen years! Good God in Heaven, you’re having me on.” Hannigan stood his ground.

“Who made the complaint?”

“Somebody called Fanny Armstrong. Said she was a movie director. Called from some kind of ‘solar movie tent’ over by the U.N.” In answer to the look on Fitzgerald’s face, he said, “What she said, Sarge.”

“Gimme the number.” Hannigan handed over the paperwork and made his way to the coffee pot. As he was stirring his two sugars he heard Fitzgerald make contact.

“Mrs. Armstrong…Ah, sorry, then. Miss Armstrong…Oh, Mizz, is it? Well, then, Mizz. This is Sergeant Fitzgerald from the 17th precinct. I understand you are looking for, what was it, ‘Global Warming’?…Yes, yes…I see…Yes, quite understandable. But Mizz Armstrong, what puzzles us is why you waited for—what was it?—seventeen years before making a complaint?…Ah, you do, then…Yes…No, I see that could be a problem…No, Mizz, I don’t mind holding.”

Hannigan placed a coffee on Fitzgerald’s desk. The Sergeant took a sip then covered the microphone with his hand. “Mizz Armstrong is taking a call from His Eminence Ban Ki Moon.” More coffee.

“You were right about that tent-thing, Hannigan. This Global Warming, as far as I can make out, is to be featured in some movie premiere, a world-wide broadcast. They’re in a panic because their star can’t be found. Fetch me a new pen, now, would you? This one is dry.”

Fitzgerald sipped at his coffee and settled back to wait, but not for long. “Ah, Mizz. Mr Moon doing well, is he?…Good, good. Remember me to him, would you? Mention parking tickets…Don’t worry, he’ll know. Now, we need some facts before we can help. For instance, what does this Global Warming look like?…Uh huh…Yes…yes…Are you sure, Mizz?…Well, the reason I ask, Mizz, is that the description doesn’t match anything that we…True, Mizz, true. Just a second. Let me ask one of my men who knows the area well.”

“Hannigan, there. Young Mizz says that Global Warming can be found in the temperature record. Just you have a look at it. She says that it will show as a steady, dangerously increasing line, starting from about 1970.”

“Nothing like that here, Sarge. The series seems to be going down or holding steady, and has been for a long time.”

“Did you check the outer boroughs? Be careful with Staten Island. Being that close to the ocean can hide changes in diurnal temperature variations.”

“No, Sarge, nothing. Not anywhere in the world. No record of a Global Warming answering to her description.”

“You heard, Mizz?…No. I assure you our records are quite complete…Um, hmm…Yes. Well, let’s put it this way. How do you even know this Global Warming exists? You’ve never actually see it…Michael Mann? No, Mizz, I have not. Christopher Nolan?…No, sorry. Me and the Mrs. prefer quiet evenings at home…I see…That is a lot of movie stars you have there…What is the name of your film?…The Age of Stupid, is it?…About how people ignore Global Warming? Perhaps they should, since it doesn’t seem to exist?…No, sorry; just a wee joke. Don’t you worry, Mizz. With all those celebrities involved, nobody will even notice that your Global Warming is missing…Quite, sure, Mizz…You have a nice day, too.” He hung up the phone and said to Harrigan, “Another case solved.”

Hannigan went to his desk to finish his paperwork, glad he didn’t have to go out. Fitzgerald again crossed himself, closed his eyes, and said a silent prayer to St. Genesius. “Save us from celebrities if you can. They’re a nervous bunch.”


Summary Of Statistical Evidence Against Global Warming

One of global warming's more prominent supporters.

One of global warming’s more prominent supporters.

This is from my Classic Posts page, but seems appropriate to highlight in the wake of the latest gloom and doom from the IPCC.

If you’re looking for just one thing (statistically oriented), see the series on how to reconstruct and homogenize temperature series. The IPCC’s pictures, in particular their “confidence bounds” are too narrow, far too narrow. Yet another instance of the old “parameter-based” view of statistics (parameters reified as reality) and the new, predictive approach (where observables rule).

Meaning—as I’ve been saying for years—these fellows (and fellowettes!) are far too sure of themselves. If you’re on a desert isle, you can surf over to the same CP page (woefully behind in updating) and look at the statistical articles.

On the other hand, if you’re like me, you might skip reading everything. Is there any political will left for action on global warming? You can only scream so long before the fear of your audience turns to boredom and, finally?, to hostility. I find the whole area tedious; my heart sinks each time I see some earnest true believer trying to explain how the sky is falling, even though year upon years of observations shows it rising. And if you don’t like that screwy backwards metaphor, I’ve got plenty of others.

The number of blown forecasts have been so many you’d guess climatologists would go into hiding instead of trumpeting how accurate are their beliefs. I can’t even be upset with them, the poor dears. Do you know how gut wrenching it must be to even contemplate admitting to the world—the whole world which has been following your every word—that you have been wrong? Imagine the loneliness they’ll soon feel. Sad.

My eyes now glaze over, almost literally, whenever I see yet another whatever-they’re-calling-global-warming-this-week story. Maybe yours will do the same when scanning these old posts. I wouldn’t blame you.

So, this post, and tomorrow’s, may well be the last you hear from me on this subject.

Global Warming & The Environment

The BEST project: I (what it means), II (methods), III (politics).

How to think about time series (temperature example). Part I, II, III, IV, V.

What Probably Isn’t: Heat Waves and Nine Feet Tall Men: Prelude, I, II. 1 in 1.6 Million Heat Wave Chance: I, II

A Citizen’s Guide to Global Warming Evidence, Use And Abuses Of Decision Analysis: Global Warming Example, The Data Is The Data, Not The Model

Bad Astronomer Does Bad Statistics: That Wall Street Journal Editorial

How To Cheat, Or Fool Yourself, With Time Series: Climate Example, Anthropogenic Forcing Signals Not Significant?

What is—and what isn’t—evidence of global warming, Overview, I, II, III, IV, V, VI

The EPA, Dust, And The Ecological Fallacy Example: Criticism of Jerrett et al. CARB PM2.5 And Mortality Report

Do not smooth times series, you hockey puck! I, II, III

Climate Model Uncertainty: I, II

Homogenization of temperature series I, II, III, IV, V

Hurricanes have not increased: misuse of running means I, II

Proper statistical description of temperature (parameter-based versus predictive statistics): I, II.

Lewandowsky’s Faked Moon Landing

« Older posts Newer posts »

© 2015 William M. Briggs

Theme by Anders NorenUp ↑