William M. Briggs

Statistician to the Stars!

Page 146 of 408

Drug Companies Tweaking Results To Produce Publishable P-values?

A person who Nature is calling a “whistle blower” has written a brief confessional in the British Medical Journal admitting to what might be termed statistical fiddling at a “major” drug company. (The just-over-one-page article requires a subscription to view.)

It is always difficult to trust fully any gossip which is reported anonymously, for if a man can hide behind a letter he may say anything, even that which is not so, without fear of reprisal. Too, the person or organization who publishes the gossip has no way of verifying it.

Now Mr X, if we may call him that, speaks of a company’s post-FDA-approval observational studies on drugs, studies whose primary purpose are to tout the drugs. “Patented Xlimicorconaphil is better than the generic! Ask your doctor if Xlimicorconaphil is right for you. Hint: it is.”

Mr X criticizes the statistical methods of these observational studies. He claims, “the truth is that these studies had more marketing than science behind them.” Worse:

Since marketing claims needed to be backed-up scientifically, we occasionally resorted to “playing” with the data that had originally failed to show the expected result. This was done by altering the statistical method until any statistical significance was found. Such a result might not have supported the marketing claim, but it was always worth giving it a go to see what results you could produce. And it was possible because the protocols of post-marketing studies were lax, and it was not a requirement to specify any statistical methodology in detail. On the other hand, the studies were hypothesis testing (such as cohort studies, case-control studies) rather than hypothesis generating (such as case reports or adverse events reports), so playing with the data felt uncomfortable.

The dreadful, should-be-banned term “statistical significance” means a publishable p-value, i.e. one less than the magic, never-to-be-questioned number, a number given to us (rumor has it) by Merlin himself. The number is sacrosanct, it is written into the law. Studies which cannot produce the required number are shunned. Those that find wee p-values are glorified.

Now especially in observations studies, this desirable creature, the wee p-value, can always be found, as long as one is willing to rummage around the data for a sufficient length of time. Mr X claims that is what his drug company has done. He appears to think this practice unusual and a bit shifty. Shifty it may be, but unusual it is not. It is not confined to observational studies, but appears even in designed experiments. And this is to be expected when success is defined in terms of p-values.

Statistics in this way is like a machine into which is fed data, a crank is turned, and out pops a rotten egg or one made of gold. Turning the crank longer increases the chance of gold. Success is trivially identified, but so is failure. The process requires no thinking (except by the nameless mechanics who keep the machine running).

Mr X also claims:

Other practices to ensure the marketing message was clear in the final publication included omission of negative results, usually in secondary outcome measures that had not been specified in the protocol, or inflating the importance of secondary outcome measures if they were positive when the primary measure was not.

Which sounds like standard politics. But I wonder. How often do drug companies try to hide negative results? Truly negative, I mean. Like discovering that widows who eat Xlimicorconaphil stroke out at rates exceeding the general population? What happens when this aberration finally outs? Smells like jail time.

Instead it’s more likely that the kind of “negative” results Mr X means are slight increases of slightly higher blood pressures in some subset of a subset of the population of those who take especially high doses of Xlimicorconaphil. Not a good thing, but not as awful as death or disfigurement.

Anyway, those negative findings are just that: findings. Produced using the same questionable statistical procedures as the positive findings which Mr X isn’t so keen about. How robust are they then? Probably not very.

The truth for most drugs is usually something like this: Xlimicorconaphil was found, via the usual FDA process, to be marginally better than the generic in some subset of the population. Xlimicorconaphil produces slightly different side effects, or of different intensity or frequency. The drug company, having to recoup its investment, takes this information, dresses it up, and sells the pill as New and Improved!

Nothing shady about this, especially in our all-marketing-all-the-time culture where such behavior is expected of everyone. The real worry is if doctors cease being skeptical gatekeepers.


Thanks to Brad Tittle who suggested this topic.

2012 UEFA European Football Championship And What Probability Is

A pair of statisticians have calculated that, as of 10 June, the probability that Germany takes the 2012 Euro Cup is 26.8%. Russia, they say, comes in with the next highest probability at 17.5%. Poor Ireland was given only a 0.1% chance to win. (I am rooting for Germany and Poland.)

Magne Aldrin and Anders Løland say that

The probabilities are found by simulating (“playing” on a computer) all remaining matches numerous times. The remaining part of the tournament is “played” 10,000 times and from these simulations we calculate all the teams’ chances of winning the Championship, their group, to reach the second round etc.

In other words, they have some model which takes in information selected by the pair and out spits the probabilities. The natural question many want to ask is: are these the right probabilities? And the answer is: yes, as long as they haven’t made any errors in the calculations or in typing out the results for press.

Now compare the MAAL (authors’ initials) probabilities with those produced by the patented Briggs Soccer Model-O-Matic. This model says that Germany has a 6.3% chance of winning. But it also says that Russia has a 6.3% chance; and so does Ireland. So do each of the 16 teams; indeed, they all have equal probability of winning. The natural question now to ask is: are these the right probabilities? And the answer again is yes.

This sounds nuts. How can the MAAL and the BSMOM probabilities both be right? Shouldn’t there exist, somewhere out there, one final set of probabilities which we can discover, a true set which we can all know? Well, no and yes.

The MAAL probabilities are true assuming the model which produced them is true. The BSMOM probabilities are true assuming the model which produced them is true instead. Which model is truly true? Well, if we knew that there would be no use deriving different models. And the MAAL and the BSMOM aren’t the only models in contention: each bookie has a different one, and probably so do other statisticians.

Consider what is happening here. All probabilities (indeed, all statements of knowledge) are conditional on propositions which are known or assumed to be true. These propositions are what the models are. I mean, any logically different set of propositions, which includes observational propositions (e.g. “Germany won so many games last year”), make up what a model is.

For example, the BSMOM model, i.e. its list of propositions, are this: “There are 16 teams in contention (and here is the list), just one of which can win.” From assuming this model is true we deduce the chance that Germany wins is 6.3%, etc.

The MAAL model’s list of propositions is longer and more complex, but in the end it is just a list of propositions which we assume is, or really is, true and thus let us deduce the chance each team will win. All statistical models are the same: mere lists of propositions we assume are true, or really are true, from which we deduce probabilities of events.

There is still a sense that the MAAL is a better model than the BSMOM, however. We have the feeling that, for any situation, there exists what we can call the Omniscient Model. This is a list of propositions which are true and which lead us to deduce that the chance that (in this situation) Germany wins is either 0% or 100%. Since the propositions which make up the OM are true, the chance that Germany wins is 0% or 100%.

There always exists an Omniscient Model, even for quantum mechanical events; the trouble is that, especially for events on the smallest scale, we rarely know what this model is (and for QM it seems we cannot know). But here is where the sense that the MAAL is better than the BSMOM arises: the closer any model’s propositions are to the propositions in the Omniscient Model’s list, the better that model will be. Surely, our gut tells us, the BSMOM is father from the truth.

Our gut is probably right, but it’s relying on its own model which says roughly, “In my experience, models like the BSMOM rarely produce useful information; while models like the MAAL are better.” We can only confirm our gut after the fact, but measuring how close the probabilities of each model are from the actual outcomes. Those measures become yet another proposition which feeds back to our gut (or into a meta-model in a formal sense) so that when we meet MAAL-like and BSMOM-like models in the future we’ll have an idea which to prefer.

Update Fixed asinine mistake and typo. Thanks Uncle Mike and Stephen D!

In Praise Of Narrow Minds


Jonah Goldberg, in his The Tyranny of Clichés relates how Andrew Sullivan quoted approvingly a story from an old book whose moral was that one should not believe morals of stories from old books, particularly those books which advocate restrained behavior. Ancient and merely old morals are held only by those who have narrow minds, you see.

“MissionGathering Christian Church IS SORRY,” screamed a billboard noted by Timothy Dalrymple, “for the narrow-minded, judgmental, deceptive, manipulative actions of THOSE WHO DENIED RIGHTS AND EQUALITY TO SO MANY IN THE NAME OF GOD.” The folks responsible for the billboard were displeased that North Carolina had voted against “gay marriage.”

Sullivan, the angry parishioners at MissionGathering, and most of the rest of us moderns just won’t tolerate the narrow-minded. The term is everywhere one of opprobrium, when it should be one of approbation.

What irritates many, and what seemingly gives justice to the insult, are those narrow-minded folks who have chosen to believe what is absurd. The Van Jones 9/11 “truthers” who insist that a mustachioed Dick Cheney personally planted thermite charges in the World Trade Center; the Oliver Stone groupies who are convinced that JFK was murdered by Marilyn Monroe using a gun concealed in her brassiere; or abductees who believe they have been taken forcibly on a three-hour tour of the upper atmosphere by grey aliens and who describe in nauseating (yet loving) detail the precise placement of probes into their persons.

If you think these people do not know all the facts, if you believe that all you have to do is to present to these stalwarts all the relevant evidence and that they will then recant, then it is you who are mistaken. These passionate people know everything about their subjects, more than you could hope or ever want to learn. They have at the ready angles, trajectories, time-lines, weights and measurements, the various chemical considerations of steel, family trees, all woven into theories more intricate than any Byzantine tile.

And this is usually the case when a man believes something which is false. His mistake is not in collecting clues, but in how he ties these threads together. This is why naive-minded programs to “educate” the mistaken fail, and will always fail. Education is in these instances like attempting to untie the Gordian knot, like attacking the enemy where he is strongest and most fortified, when what is needed is to cut the thread which holds the knot to the man’s desires. What bothers us about these people is not their narrow minds, but their embrace of error.

Now there is nobody as narrow-minded and as dogmatic as a mathematician. This is a man who just will not open his mind to hearing about new methods to square the circle or to show that two plus two is sometimes not four. He will rebuff, sometimes angrily, arguments which claim triangles have four sides. What a judgmental bigot! It is as if this man is in thrall to a religion, who actually has hold of Truth and believes it come what may. A truly closed mind.

We should all have minds as narrow. To possess and hold Truth—and not to be talked out of it because of faulty, frivolous insults, or because many have decided to be against you.

Just think: If you have ever been called narrow-minded then you know that your interlocutor does not want to broaden your scope, to open your mind, to make it fuller, more “accepting.” What your adversary wants is for you to change your mind, to believe differently but just as narrowly as he does, to reject what you previously believed.

The parishioners at MissionGathering have made up their minds that it is certainly the case that men should be allowed to marry men, and that all should embrace this new custom. This is a very narrow view, just as narrow as dogmatically insisting that circles have no corners. This being so, these parishioners cannot really be angry that their adversaries have narrow minds. They are incensed only because somebody believes differently than them.

And the same is true for Andrew Sullivan who would have you reject those books with which he disapproves, but who would have you embrace those books with which he approves. Sullivan would have you narrow your mind, to become as judgmental as he.

Of course, it might be true that it is Sullivan who holds the truth and that the ancients and our elders who were mistaken. In that case, we should narrow our minds, and change them in the direction Sullivan et alia point. But we have one large clue that Sullivan and others like him are wrong. And that is they can do no better than to employ logically absurd abuse when confronted with opposition.

“Say, That’s A Nice Hat”; Or, I Get Touched

I should have known better. Me. Mr Street Smarts. A man who has actually vacationed in Detroit. On purpose. A guy who has walked all over New York City, trudged up and down Ellis street in San Francisco without flinching, and ventured through Whisper Alley amidst throngs of drunken Marines.

Yet none of that mattered yesterday.

Now I get a lot of compliments on my hat, a white Panama (pictured above and purchased here). On Saturday for instance I was stopped twice and told how pretty. My theory is that it isn’t so much the hat that is remarkable, and even less so the wearer, but that people like to see an adult wear an adult hat while suitably suited. It is so rare a sight, particularly in some areas of this grand country, that it is worth celebrating even if the spectacle is carried out by a fellow like yours truly.

So it wasn’t a surprise to hear a voice yesterday (Sunday) call out, “That’s a nice hat.” There were other words, too, but I didn’t catch them. The man who spoke, however, stopped and turned. Which out of a misguided sense of politeness made me do the same.

“Hey, you don’t remember me, do you. I saw you coming and even complimented your hat. But you don’t remember me.”

It was at this point, early on, that I made the fundamental mistake. I spoke. “No, sorry I don’t.”

“We met years ago. I’m the brother of one of the women you work with. You remember. Black woman you work with.”

“Deborah?” I volunteered (not her real name). The first name to pop into my mind. And I didn’t think to question why he, a black man, had to tell me his sister was also black. Nor why he made me guess.

“Yeah! Deborah. We met years ago.” He proffered his hand. Which I shook. “When’s the last time you saw Deborah?”

“Oh, must be years ago.”

“Then you haven’t heard. She had a stroke, man. Paralyzed below the waist. Has to use a walker to get around.”

It was at this point that, for any intelligent person, Klaxons would have been going off. But I remained sweetly oblivious. How the hell could she be paralyzed from the waist down and use a walker? “Oh no! Can she still teach?”

“Oh, yeah, she can still teach.” He also, in the course of this interview, reminded me several times of how I had ignored him, how he had liked my hat, and that I should have remembered him.

He was good. He already had me at this point, but he didn’t go right for the kill. He wanted to secure the hook a little deeper first. So he talked of this and that. Got me to say a few things. Then he began.

“I bought this car. That’s why I’m in this neighborhood. I’m parked over there. I just bought this thing. But the gas gauge is broken. I thought I had a full tank of gas. I thought I had great gas mileage. But it’s out of gas.”

This was another chance for me to make a break. I knew then it was a touch, but I had lingered so long I couldn’t figure a way to make a decent exit. Which is absurd, because once I knew it was to be a touch, I could have said and done anything in good conscience.

“I had to search forever to find a gas station. But there’s none in this neighborhood.”

I stupidly was still playing along and pointed, “Except for one up…” An answer he nearly simultaneously mimicked. As if he knew where the station was.

Then came the tale of a deposit on a gas can and lack of cash. I said he was good and he was. He didn’t ask for the money but waited patiently for me to volunteer it. I did. Five bucks and some change. Just to end the thing and escape. As I forked it over he asked if I could get more, go to a cash machine or something. And then I told the truth. “I don’t have my wallet on me.”

Ever the pro, he did not try to rush off. He tried to get my contact information so that he could “return” the money. I insisted he just keep it—I didn’t want him having my name, which I had only then realized he never knew, never used, and never asked for.

As he left I marveled how very like a storefront psychic he was. All the information about Deborah, including her name, the gas station, teaching, he had got from me. He just parroted it back and wove it into a plausible story. If I weren’t paying attention, when I later recalled the conversation I might have convinced myself that he really was Deborah’s brother. But then maybe I would have felt like I had done a good deed, instead of realizing what a big dope I was.

We Are All Eugenicists Now: New Test Identifies 3,500 Genetic “Faults” In Fetuses — Update

See the crucial update below. I was assured, several times, that my second son was to be my first daughter. The doctors who told me this had performed certain tests, you see, and these unambiguous, quite scientific tests revealed the absence of that which makes a girl into a boy.

Now this was just more than a score of years ago, a time when medical science was little removed from bleeding and letting diseases run their course without the application of massive amounts of money. But this is hindsight. At the time—in the moment—all of us had high confidence that the tests were accurate. Just as we think the tests that exist now are accurate.

There is in statistics a hoary old mind-twister which we use to tease first-year students. It runs like this. You have just been tested for a disease and been told the test was “positive”, which means the test says you’re in for it. The question is: given this information, and some evidence about the disease and test, what is the probability you actually do have the malady?

We teachers delight when students say “100%”, “99%”, and other high numbers. This makes us happy because these answers are as wrong as can be—and because we love to shock our pupils. The real answer is usually near 10%, or even less. Why? Because tests are imperfect and most diseases are rare.

It isn’t just the rarity of the disease that causes this result. It’s the imperfection of the tests. Even the best ones aren’t that good; certainly not when used in isolation and without the benefit of further tests and other diagnostic markers. Every doctor knows this.

But patients, and students of probability, do not. When they hear a test say that “Things are this-and-such” they believe that things really are this-and-such.

It was reported this week that science, the answer of all things, had invented a new battery of tests which would scan a fetus for, count ‘em, 3,500 genetic “faults.” We are told that some of these “faults” “include Down’s syndrome and cystic fibrosis.”

Pause here, Mr or Mrs Relativist. The word fault is objective; it implies a universal, or at least a universally agreed to, standard. A fault is that which we all agree is a defect, a thing undesired and undesirable. It is that which is to be weeded out. Culled. Or erased.

We would, I hope, react with abhorrence if a scientist had claimed to have developed a test which would identify those—living, walking, breathing—individuals who were deemed genetically imperfect. Which is to say, impure. Because even if the act of identifying those of us who are less than (racial) perfection is not tied to any program to eradicate these impurities from the gene pool, the suspicion that efforts along these lines could at any moment organize is surely historically warranted.

But if it is instead announced that new and improved methods have been invented for genetic weeding of the womb, well, pop the champagne corks! The glorious forefronts of science and all that. Nobody says it, but in these stories the implication is always that killing the genetically faulty before they make their entrance is good for all of us. We would not want to be burdened with that which is less than perfect. Incidentally, good, ladies and gentlemen, is another one of those objective, universal standards.

Previous genetic tests could identify, at best, a few dozen imperfections. Last week’s news pushes that number much higher; and presumably this figure will grow by an order of magnitude in a decade. This means that the fault-lines are ever widening. The inescapable conclusions is that our idea of genetic perfection is growing narrower, ever narrower.

Circle back to the beginning and recall that medical tests are imperfect. England’s Lord Robert Winston remembered. He warned that the new tests would raise “many ethical questions.”

This is British understatement for the truth that many non-mothers-to-be relying on these tests will kill off perfectly healthy fetuses which they believe to be defective. To be, that is, genetic inferiors. In that group of women willing to kill their fetuses and who trust testing, the rate of killing-in-error will be high, because the chance of an error can only increase with the number of items tested.

This, dear reader, is a mathematical truth. The more tests there are, the greater the chance of an error. It is also true that newer tests are more prone to mistake, meaning that as the number of tests increase the chance of an error in killing increases much faster than you would think. Until, that is, it becomes almost certain.

But then, it always was certain.

Update In the same article which quote Lord Winston, we also hear from Professor John Harris, director of the Institute for Science, Ethics and Innovation at the University of Manchester, who has this to say

No potential being has a right to become an actual being — abortion is not a “wrong” to the individual because the individual in question will never have existed.

We would be negligent and reckless if we paid no attention to the health care of future generations and future people. The ability to protect future generations from terrible conditions that will blight their lives seems to me to be an absolute moral responsibility and a duty that we should not shirk.

This is full-blown eugenics; no other way to spin it. “Protecting” human beings from faults by killing them is an argument only an academic could love. And could say and get away with.

I’ll agree with the good professor about one point, at least partially. “No potential being has a right to become an actual being.” Many eggs and most male gametes, which potentially could have been combined to form a human being, haven’t a “right” to be so combined; thus there is no right to exist in this sense.

But the bad professor is quite wrong—he is committing a silly fallacy—by moving from this to saying abortion isn’t “wrong” (notice the scare quotes) because, he is tacitly arguing, the fetus is not an individual. But that, dear ones, is the very point at question: whether the fetus is a human being. It is certainly an actual being of some kind: it is not a potential being. It is a human being, too, at the earliest stages of its life. If you say not, then what exactly is it? At what point and just how—exactly, please—does it become a human being?

The real injustice of it is that even though Prof. Harris argue so badly, he is still collecting a paycheck. For arguing so badly. Yeesh.

Update Worth noting.

« Older posts Newer posts »

© 2014 William M. Briggs

Theme by Anders NorenUp ↑