Post-Approval Drug Failures: More Reasons To Eliminate P-values & Parameter-Centric Methods

Some comments on the JAMA: Internal Medicine article “An International Perspective on Drugs for Cancer: The Best of Times, the Worst of Times” by Richard Lehman.

Drugs increase fast, and there is “a widespread misperception that newer generally equals better, regulatory authorities have the unenviable duty to adjudicate on which treatments should be made available for use.” That newer is better is the Progressive Fallacy.

Here’s the meat (my emphasis):

Most new cancer drugs are approved through the FDA’s accelerated approval process, which allows for drugs to be approved faster based on surrogate end points that are thought to reasonably predict a drug’s clinical benefit. However, surrogate end points are often poorly correlated with survival, and little is known about how they correlate with other patient-centered outcomes such as quality of life. Hence, in return for allowing quicker access to the market through the accelerated approval program, postmarket confirmatory trials are required to verify clinical efficacy.

Use of surrogate end points when the surrogacy is forgotten or masked is one version of the Epidemiologist Fallacy. That fallacy is the ecological fallacy coupled with the use of significance testing or parameter-centric methods, which falsely imply causation. Another version of the fallacy is when surrogate “x” measurements are used. This happens all the time in medicine. See for example, PM2.5 studies.

Those methods wildly exaggerate evidence in the favor of researchers’ pet hypotheses, as discussed in this paper.

STATISTICS MUST RELEASE ITSELF FROM PARAMETERS AND TESTING!

Surrogacy has consequences. It allows cheating.

The study by Gyawali and colleagues examined 93 approvals that were granted on the basis of a variety of surrogate outcomes. The authors demonstrated that outcomes used in postmarket trials often apply the same surrogates as were used in the preapproval trials. The authors found that only 16% (n = 15) of approval confirmations were based on clear evidence of improved overall survival.

The second article gave similar dismal results, but about the even weaker outcome of response rate (defined as any improvement). But then Lehman says “These articles serve as a reminder that the accelerated approval pathway is a permissive process that tolerates nonrandomized trial methods and a variety of outcome measures that bear an uncertain relationship to patient benefit.”

A reminder that “randomization” provides nothing to a study and can even be harmful. Control is king; control is paramount. Such as in treatment assignment.

Such findings build on a growing body of work, which in aggregate, demonstrate a postmarketing evaluation process that is serving neither patients nor society well. First, the surrogates used for accelerated approval are poor predictors of clinically meaningful outcome. Second, study designs of post-marketing studies may not provide adequate information about comparative efficacy because some studies may not use appropriate control groups or rigorously account for treatment assignment bias

Here’s the main thing (my emphasis):

Given the unsatisfactory nature of these surrogate outcome measures, it is of the greatest concern that the studies by Gyawali et al and Chen et al show that most of the drugs given accelerated approval have not been shown to improve overall survival in postmarketing trials.

Not only does surrogacy not well predict survival, it is again the case that hypothesis testing and parameter-centric methods always and necessarily exaggerate evidence in favor of whatever hypothesis is advanced.

Proof? Besides that must-read paper, the Chen study says “many drugs that are approved based on [response rate] do not have any substantial drug activity”, which they discovered in post-market trials. Even that finding is based on classical methods (parameters, testing), so the real awfulness of the situation is even worse.

Even NPR is admitting this: “For example, Genentech’s Avastin, or bevacizumab, won accelerated approval to treat the deadly brain cancer glioblastoma, but the drug did not extend the lives of patients in a follow-up study.”

Incidentally, is it only a coincidence that modern drugs sound like they’re named for demons? I summon thee, Great Bevacizumab, to remove this lump!

Relevant to this is this talk on medical nihilism.

Stegenga argues that many medical treatments either fail to achieve their intended goals or achieve those goals with many negative side effects. Stegenga argues that the approval process for pharmaceuticals, for example, exaggerates benefits and underestimates costs. He criticizes the FDA approval process for approving too many drugs that are not sufficiently helpful relative to their side effects. Stegenga argues for a more realistic understanding of what medical practice can and cannot achieve.