Introduction
Marchmain Pharmaceuticals distributes profitizol, said to cure the screaming willies. To assess that claim, Marchmain assembled volunteers suffering this dread malady. Each provided his address, which was used to count the number of pharmacies within a five-mile radius of where each person lived.
Most pharmacies sell profitizol, and the more pharmacies near a volunteer the more opportunities there were for volunteers to purchase profitizol, and then eat it. Marchmain asked a statistician whether living near more pharmacies correlated with being cured, which resulted in the announcement, “Taking Profitizol Increases Chance of Screaming Willies Cure.” Marchmain’s stock subsequently soared.
I know what you’re thinking: Surely nobody would claim that merely living near a pharmacy is proof that profitizol “works.” That’s where you’re wrong, friend. Not only do people routinely make claims of this very sort, entire scientific fields and numerous government bureaucracies positively rely on this trick for their existences.
The epidemiologist fallacy occurs when an epidemiologist says or implies X causes Y, but when the epidemiologist never actually meets, measures, or monitors X, though everybody pretends he has.
The “X”s and “Y”s are placeholders, stand-ins for common English propositions like X = “It is cloudy” and Y = “It is raining.” If X is correlated1 with Y, it means our understanding of the values (states) of Y change according to changes in the values (states) of X. If X = “It is cloudy” it is likely to Y = “rain”, and vice versa. Cloud cover and rain are correlated. But if, no matter what value X takes, our uncertainty in Y remains unchanged, then X is uncorrelated with Y; knowing X is irrelevant to our knowledge of Y; classical statisticians say X is independent of Y.
For Marchmain, Y = “Person cured of the screaming willies” and X = “Person ate profitizol.” But while Y was measured, X was never observed. Yet Marchmain still announced that X was correlated with Y. How?
Via the epidemiologist fallacy. Marchmain invented a W, which is not X, but which was kinda sorta like X—well, loosely like X, in a vague way, if you squinted—-and then swapped X for W. The statistician modeled the correlation between W and Y, but announced that this correlation was between X and Y. W was forgotten. Since everybody wanted news of X and Y, they failed to see it was W and not X over which the fuss is being made. Government grants were awarded.
Examples
Global warming causes cataracts in babies
The peer-reviewed paper “A Population-Based Case-Control Study of Extreme Summer Temperature and Birth Defects” appeared in the journal Environmental Health Perspectives (2012 October; 120(10): 1443–1449) by Alissa Van Zutphen et alia. It purportedly investigated birth defects in New York residents (the Y) and heat waves during pregnancy (X), which were claimed to increase in frequency and severity once global warming finally strikes. “We found positive and consistent associations between multiple heat indicators during the relevant developmental window and congenital cataracts [in newborns]”. Various statistical measures of correlation were attested to, and if the reader wasn’t careful she would decide to stay out of the heat lest her unborn child develop congenital cataracts.
But exposure of women to heat during their “relevant development windows” was never measured on any woman. There was no X. But there was a W: the daily air temperature at “18 first-order airport weather stations”. Women were assigned the temperature at the stations closest to where they listed their residence at the time of birth for just those days thought to be crucial to fetal development. Nobody knows where the women actually were during these days: it may have been near the assigned airport, or it could have been Saskatchewan, or perhaps in some cool building (“we were unable to incorporate air conditioner use data”). This paper was taken seriously by the press. More research is needed.
Fourth of July parade attendance turns people into Republicans
Harvard Kennedy School Assistant Professor David Yanagizawa-Drott and Bocconi University Assistant Professor Andreas Madestam wondered how it could be that so many innocent Americans turned into Republicans (their Y). They suspected Fourth of July parade attendance (X). Exposure to raw, unfiltered patriotism would take its inevitable toll and cause people to turn wistful at the mention of Ronald Reagan. They speculated, “Fourth of July celebrations in the United States shape the nation’s political landscape by forming beliefs and increasing participation, primarily in favor of the Republican Party.”
It was widely reported that X caused Y. Only it wasn’t so. Yanagizawa-Drott and Madestam instead created a W. They gathered precipitation data from 1920-1990 in towns where study participants claimed to have lived when young. If it rained on the relevant Fourths of July, the authors claimed the participants did not go to a parade, because they assumed all parades would be canceled. If it did not rain, they claimed participants did go to a parade, because all towns invariably have parades on clear days, and if there is a parade one must attend. Nowhere was actual parade attendance (X) measured. And just think: if their hypothesis were true, San Francisco would be teeming with Republicans because it almost never rains there on the Fourth of July.
Air pollution causes heart disease
Yours Truly was involved in a critique of a study submitted to the California Air Resources Board (CARB) which claimed to have discovered a correlation between air pollution (X; particulates of a certain size) and heart disease (Y). A weak, barely there finding of statistical “significance” was enough to embolden CARB to create new and enhance old air pollution regulations in order to “save lives.” Yet X was never measured.
At a very few places, particulate measures were taken for a limited time. The air pollution in these places was then crudely extrapolated to areas in which it was not measured. Finally, the extrapolated air pollution nearest the address of the study participants (where they lived at one time, ignoring moves) was taken as the exposure; this was their W. Nobody knows how much air pollution anybody was actually exposed to.
The sequel to this story is fascinating. I submitted written critiques where they were discussed at a CARB meeting. One panel member thanked me, called me learned, and took my criticisms of the epidemiologist fallacy seriously. But it was judged that—and here you must laugh—because the fallacy was so common that it led to many results referenced by CARB, that this current study was no different. And therefore acceptable.
We could continue examples indefinitely. It has become rare not to see it used, particularly in faddish research, such as the evils that awaits us when global warming eventually strikes.
Over-certainty Guaranteed
Scientists get away with this fallacy because often W and X are correlated themselves, or are thought or hoped to be. Yet logic insists we must necessarily be less sure of the relationship of X and Y than we are of W and Y. If W is correlated to X, which in turn is correlated to Y, then W will be correlated to Y. But the correlation between W and Y is not the same as that between X and Y. Overconfidence abounds.
It’s easy write papers invoking the epidemiologist fallacy. All it takes is an afternoon, a computer, and a wild theory. And with the brutal competition to publish, to be cited, and to win grants, there just isn’t any way to stop researchers misbehaving. Your only hope is to cease blindly trusting science reporting and government agencies.
And now my favorite line, beloved by activists everywhere: it’s worse than we thought! Not only do scientists incorrectly swap W for X, sometimes a Z and not a Y is measured. Yet the story is still X causes Y. Sociologists have the most fun with this. Take the peer-reviewed article “Red Light States: Who Buys Online Adult Entertainment?” (Journal of Economic Perspectives, Volume 23, Number 1, Winter 2009, Pages 209–220) by Benjamin Edelman at the Harvard Business School who claimed red states consume more pornography than blue states. He implied conservatives are naughtier than progressives. Yet no individual’s consumption or political views were ever measured.
———————————————————————————————-
1Statisticians have a formal definition of correlated which I do not use here. Correlated merely means what it says above: an association.
Buy my new book and own your enemies: Everything You Believe Is Wrong.
Subscribe or donate to support this site and its wholly independent host using credit card or PayPal click here
I assume that you are aware of John Brignell’s important work in this area at Number Watch as well as his two books “Sorry, Wrong Number!” and “The Epidemiologists”. If not, may I recommend them.
Scotian,
We love him! Here’s his site: Number Watch. It’s a must visit.
I don’t have the time or even half-baked idea to share to stir things up … but… no doubt there’s ample material out there, stuff that people really worry about so much that they’ll believe anything if its presented seriously, that can serve as inflammatory examples of “Too Damn Suredness.” Topics like thus&such cause impotence, or gray hair, or hair to fall out are sure-fire winners as are themes associated with political views that are wrong, and, how this or that religious doctrine is, or was, very wrong. Those last two are sure to raise the hackles of multiple groups chock-full of people that actively seek out things get riled up over — and, most importantly, rant about on-line including scathing admonitions to NOT read the book. That’s priceless advertising ready to be got for free…the “forbidden fruit syndrome” compels people to do what, or read what, they’re told not to … so sales will skyrocket like Howard Stern’s listeners who hated him grew at the fastest pace because they just couldn’t help but hear what he’d say next (just like those that liked him). People want to get pissed off…so give’m piss to read and the sales & profits will soar! Unfortunately, I’m betting Briggs will write-up some objective, responsible, substantive academic work that even his quirkily entertaining writing style cannot possibly offset for the masses conditioned to sound-bite-sized morsels of info & a great opportunity will have been squandered.
I think the part about using W but not mentioning that it’s W you’re using is the key.
It reminds me of the scene in one of the Godfather movies (I think) where the hitman is describing what to do after shooting someone in a restaurant: let your shooting arm go to your side, walk casually out of the restaurant letting the gun slip out of your fingers, don’t look anyone in the eye on the way out, etc. Presumably, the idea was to not giving any dramatic visuals for eye witnesses to latch onto, like holding the gun in view, or throwing it against the wall.
In the same way, let the W slip from your paper casually, and like the hit man, or magician, people won’t see the important details lost in the casual moves.
As a rookie, I would have made the mistake of trying to call W an Instrumental Variable. Rookie mistake, because people would then think of “Weak Instrumental Variables”, various ways to think about how an IV might be weak, and whether W was truly uncorrelated with an error term, etc. It takes a master to let it slip from his fingers while walking down the hallway, as if it was never there.
It’s like the EPA enviromental tobacco smoke (ETS) study where they concluded that exposure to ETS causes lung cancer but they didn’t measure any exposure. People were asked about their exposure, i.e. they were given a questionare. According to the EPA, causality can be determined by taking a poll and statistically analyzing the answers. After you analyze the answers you calculate a relative risk, which is an unmeasurable stastical parameter that has no physical existence. Then throught the nagic of attributable risk they turn this into a body count which they claim actually exists. Isn’t that reification?
Serial killers are forever in search of the intense thrill of the first kill but inevitably find each succeeding one pales in comparison to the one before and none ever equal the first.
The serial epidemiologist has a nearly identical problem. Each find is of less importance and reach than the last. The return of investment within the field diminishes with each uncovered find. Sadly, none of them seem quite as satisfying as that initial water pump discovery. In fact, if it weren’t for statistics,we likely wouldn’t even be aware of the effects, let alone the causes, of the more recent finds.
Worse, the number of practitioners has increased; increasing competition within the field. Thus the urge to publish and trumpet even if the only justification is smoke and mirrors.
Benjamin Edelman at the Harvard Business School who claimed red states consume more pornography than blue states.
Perhaps he was confusing the political shading with the supposed color of the lights in the seamier districts of towns.
>>But if, no matter what value X takes, our uncertainty in Y remains unchanged, then X is uncorrelated with Y; knowing X is irrelevant to our knowledge of Y; classical statisticians say X is independent of Y.<<
In probability theory, two random variables being uncorrelated does not imply their independence.
http://en.wikipedia.org/wiki/Normally_distributed_and_uncorrelated_does_not_imply_independent
Have you taken any probability courses?
Tom,
Be sure to read everything; even the footnotes. Saves everybody grief.
The problem is that X (too much government funding) causes Z (an overabundance of garbage masquerading as science). How do I know? Let’s try and experiment: Cut X by 50% and measure the decrease in Z.
Let’s apply this elsewhere:
Living in New York City is correlated with working on Wall Street and also voting for far-left loonies. Obviously Wall Street is a hotbed of Marxism.
Owning your own home is correlated living in Mississippi or West Virginia and also with being relatively wealthy. Obviously Mississippi and West Virginia must be very rich.
Buying organic food is correlated with having graduated from college which in turn is correlated with supporting nuclear energy. Obviously, organic food stores are an ideal place to collect signatures for a pro-nuclear petition.
Walking to work is correlated with shorter commute times which in turn are correlated with faster transportation. Cars only slow you down.
A weak, barely there finding of statistical “significance†was enough to embolden CARB to create new and enhance old air pollution regulations in order to “save lives.â€
A favourite quotation of mine from “The Mortality Effects of Long-Term Exposure to Particulate Air Pollution in the United Kingdom” by the Committee on the Medical Effects of Air Pollutants” (impressed yet?):
“As everyone dies eventually no lives are ever saved by reducing environmental exposures – deaths are delayed resulting in increased life expectancy.”
But “29,000 lives saved” makes better copy than “everybody gained six months”. (both interpretations are rejected in the paper btw).
You can find it here: http://www.hpa.org.uk/ProductsServices/ChemicalsPoisons/Environment/Air/
DAV — How do you know serial killers gain diminishing rewards from subsequent kills?
Presumably some do…but also presumably others do just fine by spacing out the kills…
…but you note only the former condition…
Just out of curiosity, what area do you live in & what places do you visit–I’m just guessing those are places I & others (those that read these comments) will find, purely coincidentally, are places we don’t have any desire to visit. A statistical anomaly waiting to be formally identified…
Ken,
to answer your first question in one word: entropy.
As for the rest: I sometimes feel the same. We should all count our blessings.
nice analysis and trenchant examples. In my graduate statistics class, such a misplaced attribution of causality was also termed ignoring hidden variables …the example the prof used there was that growing taller causes your hair to grow shorter (from a correlation of hair length vs height in a class of males / females–before (or after?) long male hair was popular).
Briggs,
Be sure to note that some words have special meanings within a field of study.
Bob – Here’s another one. As a matter of fact I’m making an unproved assumption, but I think it’s on fairly firm ground.
There is a significant positive correlation, in northern USA and Canada, between the number of men in large stores wearing a particular garb (fat suit, red trousers, red hooded jacket trimmed with white fur, big boots and a false white beard and moustache combo) and levels of snowfall. The red clothes cause the snow, right?
Briggs, of course… and snowfall causes Rudolph’s nose to grow red so red clothes cause Rudolph’s nose to grow red.
Pingback: I know I’m right (?) | Climate Etc.
Pingback: Patterns and Fallacies | Startup Iceland
Pingback: Rare Win For Forces Of Sanity: Or, PM2.5 Isn’t So Scary As We’ve Been Told | William M. Briggs
Pingback: Scientists Hint That Water Perhaps Might Possibly Be Good For Kids |
Pingback: Road Rage: Paper Says Living Near Road Causes Dementia – William M. Briggs