Today’s lesson: If the government wants you bad enough, it will get you. If that isn’t already obvious, consider what befell W. Scott “Don’t Call Me Baron” Harkonen.
Just kidding with the Dune reference. Harkonen was imprisoned by the Padishah—stop that!—by our beneficent government for the most heinous crime of using a p-value which his competitors did not like.
I do not joke nor jest. Harkonen got six months house arrest for writing these words in a press release:
InterMune Announces Phase III Data Demonstrating Survival Benefit of Actimmune in IPF [idiopathic pulmonary fibrosis]. Reduces Mortality by 70% in Patients with Mild to Moderate Disease.
According to the Washington Post,”What’s unusual is that everyone agrees there weren’t any factual errors in the [press release]. The numbers were right; it’s the interpretation of them that was deemed criminal.” Post further said, “There was some talk that if Harkonen had just admitted more uncertainty in the press release—using the verb ‘suggest’ rather than ‘demonstrate’—he might have avoided prosecution.”
Harkonen followed FDA-government rules and ran a trial of his company’s drug actimmune (interferon gamma-1b) in treating IPF, hoping patients who got the drug would live longer than those fed a placebo. This happened: 46% of actimmune patients kicked over while 52% of the placebo patients handed in their dinner pails.
Unfortunately, the p-value for this observed difference was just slightly higher than the magic number: it was 0.08.
Wait! Tell me the practical difference between 0.08 and the magic number? You cannot do so. That is what makes the magic number magic. Occult thinking is rife in classical statistics. There is no justification given for the magic of the magic number other than it is magic. And it is magic because other people, Bene Gesserit fashion (last one), have said it is magic.
Therefore, p-values greater than the magic number are “insignificant.” The FDA shuns p-values that don’t fit into the special magic slot. Harkonen, holding his extra-large p-value, knew this. And wept.
I’m guessing about the weeping. But Harkonen surely knew about the mystical threshold, because he dove back into his data where he discovered that the survival difference in patients with “mild to moderate cases of the disease” was even greater, a difference which gave the splendiferously magical p-value of 0.004.
So wee was this new p-value and so giddy was Harkonen that he wrote that press release.
Which caught the attention of his enemies (rival drug company?) who ratted him out to the Justice Department’s office of consumer litigation, which, being populated by lawyers paid to snare citizens, did their duty on Harkonen.
Harkonen’s crime? Well, in classical statistics the pre-announced “primary endpoint”, what happened to all and not a subset of patients, is the only thing that should have counted. The “secondary analysis”, especially when it’s not expected, is feared and should not be used.
And rightly so when using p-values, because as long as the data set is large and rich enough, wee p-values can always be discovered even when nothing is happening, which in this case means even when the drug doesn’t work. The government therefore assumed the drug didn’t work and that Harkonen should not have used the word “demonstrated”, which it interpreted as meaning “a wee p-value less than the magic number was found.”
What makes the story pathetic is that Harkonen forgot when he got his 0.08 that the p-value is dependent on the model he picked. He could have picked another, one which gave him a smaller p-value. He could have kept searching for models until one issued a magic p-value. He might not have found one, but there’s so many different classical test statistics that it would have been worth looking.
Which of these p-values is “the” correct one? All of them!
Insult onto injury time. As Harkonen rattled his coffee cup against his mullions (house arrest, remember), his old company did a new, bigger trial on just the subset of patients who did better before. Result: more deaths in the drug than placebo group. Oops.
Anyway, maybe we should let the government, for a limited period of time, arrest and jail scientists who publicly boast of wee p-values and whose theories turn out to be garbage. Nah. Our prisons aren’t nearly big enough to handle it.
Return to this page Sunday for the highly anticipated post: Everything Wrong with P-Values Under One Roof.
Update Don’t miss the comment by Nathan Schachtman, who filed an amicus brief on Harkonen’s behalf. It’s linked below.
Thanks to Al Perrella for finding this.