Case Study Of How Easy It Is To Go Wrong

We used to talk a lot about the amygdala here: Yet Another Study “Proves” Liberal, Conservative Brain Differences (wee ps, amygdalas), Scientists Suddenly Discover Men Don’t Understand Women (wee ps, amygdalas), That Conservatives Smell Different Than Progressives Study Stinks (wee ps, amygdalas), and on and on.

I got bored with it. How many times, after all, can you write amygdala? Try even saying it three times fast.

But reader MLK sends us over the Slate Star Codex for a story that was making the rounds a while back (go there for the many, many links; my emphasis).

In 1996, some researchers discovered that depressed people often had an unusual version of the serotonin transporter gene 5-HTTLPR. The study became a psychiatric sensation, getting thousands of citations and sparking dozens of replication attempts (page 3 here lists 46).

Soon scientists moved beyond replicating the finding to trying to elucidate the mechanism. Seven studies (see here for list) found that 5-HTTLPR affected activation of the amygdala, a part of the brain involved in processing negative stimuli. In one especially interesting study, it was found to bias how the amygdala processed ambiguous facial expression; in another, it modulated how the emotional systems of the amygdala connected to the attentional systems of the anterior cingulate cortex. In addition, 5-HTTLPR was found to directly affect the reactivity of the HPA axis, the stress processing circuit leading from the adrenal glands to the brain.

As interest increased, studies began pointing to 5-HTTLPR in other psychiatric conditions as well. One study found a role in seasonal affective disorder, another in insomnia. A meta-analysis of twelve studies found a role (p = 0.001) in PTSD. A meta-analysis of twenty-three studies found a role (p = 0.000016) in anxiety-related personality traits. Even psychosis and Alzheimer’s disease, not traditionally considered serotonergic conditions, were affected. But my favorite study along these lines has to be 5-HTTLPR Polymorphism Is Associated With Nostalgia-Proneness.

It goes on like that for a few paragraph—before the big reveal, which is that it was all wee p-value nonsense.

Some 450 studies were based on the initial finding, each building on the other. Many wee p-values, many with very exceptionally wee ps. But large studies guarantee wee ps, even in the complete utter absence of causation. It is a fundamental flaw, known to all—but like confirmation bias, always believed to befall the other guy.

Long-time readers will recognized the phenomenon. One study is published, followed by another, then another, think time done on blacks or women, then a third on Puerto Rican left handers, then LGBTQWERTY Puerto Rican left handers, and on and on, each repeating the same mistakes, containing the same false premises, and all relying on p-values or parameter estimates and mistaking these for causality or real effect sizes.

Soon the mass of papers is taken as proof of the validity of the thing under study. So many smart people—and these are all PhDs at major universities with beaucoup bucks in grants!—can’t be wrong.

A criticism sometimes appears. It’s dismissed because it’s only one paper or person, weighed against all the others. People defend, cats fights ensue, insults are thrown, appeals to authority fly with abandon, et cetera.

Global warming of doom is the most prominent example, but many studies involving the differences in the amygdalas of “conservatives” and “liberals” are the same. They’d be as big in the news, too, if the “solutions” to correcting these differences were exciting to our oligarchs as the “solutions” to global warming of doom are.

SSC:

First, what bothers me isn’t just that people said 5-HTTLPR mattered and it didn’t. It’s that we built whole imaginary edifices, whole castles in the air on top of this idea of 5-HTTLPR mattering. We “figured out” how 5-HTTLPR exerted its effects, what parts of the brain it was active in, what sorts of things it interacted with, how its effects were enhanced or suppressed by the effects of other imaginary depression genes. This isn’t just an explorer coming back from the Orient and claiming there are unicorns there. It’s the explorer describing the life cycle of unicorns, what unicorns eat, all the different subspecies of unicorn, which cuts of unicorn meat are tastiest, and a blow-by-blow account of a wrestling match between unicorns and Bigfoot.

That just the way all science is. Sometimes it’s right, and sometimes it’s wrong. When it’s right, it’s helpful. When it’s wrong, it’s usually spectacularly, hilariously wrong. Not that you can get the scientists in the fields themselves to admit it. With big minds come big egos.

To support this site using credit card or PayPal click here