Bad Statistics Is Now A Crime

All p-values are not created equal, nor equally. Use one inappropriately, and you can land in jail. Just like Scott “Baron” (I’m guessing) Harkonen, former CEO of InterMune, a company which makes and pushes drugs, specifically one Actimmune, used in cases of chronic granulomatous disease (faulty lungs).

The Wall Street Journal reports that a federal judge sentenced Harkonen “to six months of home confinement this spring” and fined $20,000. Why? For using a p-value: rather, for using an un-approved p-value in an effort boost his company’s profits.

A p-value—for those who do not know, or think they do but do not—is in statistics the probability that you, if you were to repeat your data-collection experiment exactly, that you would see a certain function of your observed data larger than the function of the observed data you actually did see, all assuming that the probability model you used is perfectly (100%!) true and assuming that certain, non-observable parameters of that model equal some number (usually 0).

Specifically, a p-value does not tell you that a certain hypothesis is true or false, nor does it tell you the probability any hypothesis is true or false, nor, even, does it tell you anything about those parameters in your (assumed perfectly true) model. It is what I said it was above and nothing more.

Back to drugs. Now, all drugs before they can be unleashed on the general population have to be approved by the Food and Drug Administration, a beneficent bureaucracy of our federal government. The drugs in this process are ultimately licensed, as it were, for specific uses, such as Actimmune was for chronic granulomatous.

But doctors can use the drugs for whatever they want; they do not have to abide by the label. Even though this is so, the drug companies may not market their drugs for anything but the specific uses. But companies can seek new approval for new uses for old drugs. Harkonen did not seek new approval, but instead did some investigating on its own.

InterMune got hold of a bunch of old data, a.k.a. retrospective, and asked their team of crack statisticians to turn up publishable p-values (less than the magic, thou-shall-not-question level of 0.05) when correlating Actimmune with other diseases. They couldn’t find any.

So they (the statisticians) expanded their search and—finally!—“discovered” that patients who ingested Actimmune with “mild to moderate” idiopathic pulmonary fibrosis (another kind of faulty lung) lived longer than those who were Actimmune free.

By “discovered”, I mean the statisticians dredged the data until they were able to turn up a p-value less than 0.05—a trick which is always possible, incidentally.

Anyway, it was at this point Harkonen sent out press release, which told docs of the “statistically significant” result. Specifically, the labels of Actimmune made no new claims. This was just a press release, nothing more.

Enter the feds. Somebody at the Department of Justice, yet another beneficent bureaucracy of our federal government, noticed that the new study was merely “retrospective” and not based upon new data. They immediately charged Harkonen with the federal crime of wire fraud.

Wire fraud!

Prosecutors presented a statistician in court who said that it wasn’t possible to conclude that a drug prolonged survival simply from the results of a retrospective study. Only more rigorous trials, the prosecutors argued, could establish such a conclusion.

The feds have ruled that private companies may not talk to doctors unless the companies’ words have been first scrutinized and approved by the bureaucracy. Of course, it goes without saying, that the feds also want the companies to pay the fees that accompany the scrutinizing.

There is no greater enemy of classical statistics than I—it guarantees over-certainty—but I would not have somebody arrested for employing it. Forbidding communicating non-federally vetted research based on classical statistics would be like prohibiting the New York Times from publishing its “stories” without first securing authorization from Democrat party.

Thus far, our story is one of typical bureaucratic idiocy. It turns into a farce when we discover that

President Obama has created new institutions with the sole mandate of running trials based on softer statistical standards. Retrospective studies will be the core occupation of a new “comparative effectiveness” research agency that has $4.1 billion to conduct government studies on medical products. The results will be used to inform federal treatment guidelines, as well as Medicare’s payment policies.

At least $100 million of that $4.1 billion is being spent on promoting research results. The Agency for Healthcare Research and Quality recently paid $26 million to the PR firm of Ogilvy to “market and promote the adoption” of the findings.

Stand by for a deluge of Biblical proportions of “associations” and “correlations” and “possible connections.”

Sweet Mother of Mercy! Is this the end of sanity?