Skip to content

Bad Statistics Is Now A Crime

All p-values are not created equal, nor equally. Use one inappropriately, and you can land in jail. Just like Scott “Baron” (I’m guessing) Harkonen, former CEO of InterMune, a company which makes and pushes drugs, specifically one Actimmune, used in cases of chronic granulomatous disease (faulty lungs).

The Wall Street Journal reports that a federal judge sentenced Harkonen “to six months of home confinement this spring” and fined $20,000. Why? For using a p-value: rather, for using an un-approved p-value in an effort boost his company’s profits. Harkonen in Jail

A p-value—for those who do not know, or think they do but do not—is in statistics the probability that you, if you were to repeat your data-collection experiment exactly, that you would see a certain function of your observed data larger than the function of the observed data you actually did see, all assuming that the probability model you used is perfectly (100%!) true and assuming that certain, non-observable parameters of that model equal some number (usually 0).

Specifically, a p-value does not tell you that a certain hypothesis is true or false, nor does it tell you the probability any hypothesis is true or false, nor, even, does it tell you anything about those parameters in your (assumed perfectly true) model. It is what I said it was above and nothing more.

Back to drugs. Now, all drugs before they can be unleashed on the general population have to be approved by the Food and Drug Administration, a beneficent bureaucracy of our federal government. The drugs in this process are ultimately licensed, as it were, for specific uses, such as Actimmune was for chronic granulomatous.

But doctors can use the drugs for whatever they want; they do not have to abide by the label. Even though this is so, the drug companies may not market their drugs for anything but the specific uses. But companies can seek new approval for new uses for old drugs. Harkonen did not seek new approval, but instead did some investigating on its own.

InterMune got hold of a bunch of old data, a.k.a. retrospective, and asked their team of crack statisticians to turn up publishable p-values (less than the magic, thou-shall-not-question level of 0.05) when correlating Actimmune with other diseases. They couldn’t find any.

So they (the statisticians) expanded their search and—finally!—“discovered” that patients who ingested Actimmune with “mild to moderate” idiopathic pulmonary fibrosis (another kind of faulty lung) lived longer than those who were Actimmune free.

By “discovered”, I mean the statisticians dredged the data until they were able to turn up a p-value less than 0.05—a trick which is always possible, incidentally.

Anyway, it was at this point Harkonen sent out press release, which told docs of the “statistically significant” result. Specifically, the labels of Actimmune made no new claims. This was just a press release, nothing more.

Enter the feds. Somebody at the Department of Justice, yet another beneficent bureaucracy of our federal government, noticed that the new study was merely “retrospective” and not based upon new data. They immediately charged Harkonen with the federal crime of wire fraud.

Wire fraud!

Prosecutors presented a statistician in court who said that it wasn’t possible to conclude that a drug prolonged survival simply from the results of a retrospective study. Only more rigorous trials, the prosecutors argued, could establish such a conclusion.

The feds have ruled that private companies may not talk to doctors unless the companies’ words have been first scrutinized and approved by the bureaucracy. Of course, it goes without saying, that the feds also want the companies to pay the fees that accompany the scrutinizing.

There is no greater enemy of classical statistics than I—it guarantees over-certainty—but I would not have somebody arrested for employing it. Forbidding communicating non-federally vetted research based on classical statistics would be like prohibiting the New York Times from publishing its “stories” without first securing authorization from Democrat party.

Thus far, our story is one of typical bureaucratic idiocy. It turns into a farce when we discover that

President Obama has created new institutions with the sole mandate of running trials based on softer statistical standards. Retrospective studies will be the core occupation of a new “comparative effectiveness” research agency that has $4.1 billion to conduct government studies on medical products. The results will be used to inform federal treatment guidelines, as well as Medicare’s payment policies.

At least $100 million of that $4.1 billion is being spent on promoting research results. The Agency for Healthcare Research and Quality recently paid $26 million to the PR firm of Ogilvy to “market and promote the adoption” of the findings.

Stand by for a deluge of Biblical proportions of “associations” and “correlations” and “possible connections.”

Sweet Mother of Mercy! Is this the end of sanity?

11 thoughts on “Bad Statistics Is Now A Crime Leave a comment

  1. Do not, under penalty of federal prosecution, do what I do. Ever. Only do what I say to do.
    signed/your ever-loving beneficial gu-mint.

  2. Your final question implies that you expect sanity or rationality from your government’s manifold agencies, bureaus and departments. This strikes me as slightly insane, no disrespect intended.

  3. As the old saying goes, if you torture the data enough they will confess. All you need is a good data dredge and with a computer that’s easy.

  4. “would be like prohibiting the New York Times from publishing its “stories” without first securing authorization from Democrat party.”

    They don’t already?
    As for sanity: that ended some time ago. What we are witnessing is the birth of bizarre.

  5. P-lease Mr Briggs! You’re never satisfied. After years of reading your p-olemics against p-values, the government finally takes action against a p-value p-erpetrator. Now you say they have gone too far? P-ish tosh, says I. He’s lucky he didn’t get p-rison time!

  6. If a third party finds new uses for your drug, they can publicise it. If you find a new use for your drug, you can only wisper it.

  7. 6 months of home confinement seems like a punishment you would give to someone who is mildly dangerous, like a rioter or something. Right? I mean, I wouldn’t mind 6 months vacation mandatory confinement in the mansion of the CEO of a pharmaceutical company.

  8. Stand by for a deluge of Biblical proportions of “associations” and “correlations” and “possible connections.”

    Bah, you forgot “links”. X shall be linked to Y, and, yeah, Y to Z unto the fifth generation. And each link shall beget a second link and, verily, a third thereunto. To every link there shall be a season.

  9. This is too funny.
    The Food and Drug Administration (FDA) is too sloppy by far in establishing clear and unambiguous protocols for its drug testing. Any statistician knows that making observations under unambiguous protocols is an absolute necessity for the validity of any computed statistics, including p-values.
    How did thalidomide ever get approved? What about Phen-fen?
    To baldly assert that any claims whatever must be conducted in a Federally-approved manner, under “strict” supervision, and no publication of any sort allowed absent said scrutiny, is an abridgement of freedom of speech, freedom of commerce, restraint of trade, and a series of other violations of the Constitution.
    There are enough cases of the failure to document causation around for this whole subject to appear ludicrous.
    In fact, was there not a Nobel prize given in 1900 for curing cancer?
    To my non-lawyer mind (I practice Mathematics) there is a distinction between correlation and causation anyway.
    And what is it about 0.05? Karl Pearson set this value almost a hundred years ago. Can’t we do any better now?

  10. Mr Briggs
    You are correct in decrying this sentence. the WSJ article correctly highlights the inordinateness and obliquity of how drug efficacy certification has come to be.

    But the Fed is right on this one. Drug ‘approval’, if you believe in such a concept at all, cannot be decided on retrospective studies. In that respect, the WSJ is spinning wildly.

Leave a Reply

Your email address will not be published. Required fields are marked *