Did you know that “regions” of your brain “light up” when you think about Santa Claus or God? And that these “regions” are thought to be “associated” with various behaviors like excess emotion, schizophrenia, and other, gentler forms of nuttiness?
It’s all true. Scientists regularly stick people’s heads inside machines, ask the people to think of this or that, and then watch as the machines show “regions” of the brain glowing orange. The scientists then employ statistical methods guaranteed to generate over-confidence, but which allow the scientists to write papers which contain broad, even bracing, claims about all of humanity and of how everybody’s brain functions.
This sort of thing is all the rage, so much so that hardly a week passes without new headlines about what secrets the Whitecoat Brigade have uncovered in the brain (this week: Study shows how scientists can now ‘read your mind’).
It is therefore of great interest to us to examine this phenomenon and see what it means. I have chosen one paper which I believe is representative of the worst excesses of the field. My goal is to show you that the conclusion, as stated by the authors, and one the authors believe they have proved, is actually far from proved, is in fact scarcely more likely to be true given the experiment than it was before the experiment, and that what was actually proved was how likely scientist’s are to find in their data their own preconceptions.
Warning: I mean this critique to be exhaustive, so while I run the risk of exhausting your patience, my excuse is that the length of this piece is necessary to do a thorough job.
fMRIs and God
The paper is “The neural correlates of religious and nonreligious belief,” published in PLoS One by Sam Harris and others in association with UCLA’s Staglin Center for Cognitive Neuroscience.
The idea is plausible: perhaps different areas of the brain are used when thinking “religiously” than in thinking non-religiously. After all, different areas are the brain are activated when, say, playing baseball versus working through mathematical proofs. All we have to do is hook up people to functional magnetic resonance imaging (fMRI) machines and their beliefs will be revealed to us.
Of course “thinking religiously” is a term so broad that it would be next to impossible to distinguish between “thinking non-religiously”, so in order to have any hope of answering the main question these terms have to be rigorously quantified. How about dividing people into two groups, those that agree that “Santa Claus is a myth” versus those who hold that “Santa Claus really exists”?
No, I’m only kidding. Though Harris did ask his experimental charges these questions, as a way of calibrating his newfangled phrenological device. His actual definition was to divide the sheep and goats via statements like “The [Christian] Biblical God really exists” versus “The [Christian] Biblical God is a myth.”
Both sheep and goats were asked a bunch of questions about the Christian religion, and their brains mapped via fMRI. Differences between the brains as groups were then used to claim this result:
A comparison of both stimulus categories suggests that religious thinking is more associated with brain regions that govern emotion, self-representation, and cognitive conflict, while thinking about ordinary facts is more reliant upon memory retrieval networks…Our study compares religious thinking with ordinary cognition and, as such, constitutes a step toward developing a neuropsychology of religion.
Let’s examine the study and discover whether these bold statements are backed by the evidence.
Harris gathered 54 people who were “free of obvious psychiatric illness or suicidal ideation” (which does not imply free from psychiatric illness or suicidal ideation). The 54 were called on the phone and asked questions which “allowed us to isolate the variable of religious belief, in an effort to admit only dedicated Christians and nonbelievers into the protocol.” It would have been interesting to know what these questions were, but Harris never says (perhaps they were the same eventually used in the experiment).
The culling began:
Once we had two groups of subjects (Christians and Nonbelievers), we attempted to balance these groups with respect to 1) general reasoning ability, 2) age, and 3) years of education. We also sought to exclude all subjects who exhibited signs of psychopathology…
Forty of these participated in the fMRI portion of our study, but ten were later dropped, and their data excluded from subsequent analysis, due to technical difficulties with their scans (2 subjects), or to achieve a gender balance between the two groups (1 subject), or because their responses to our experimental stimuli indicated that they did not actually meet the criteria for inclusion in our study as either nonbelievers or committed Christians (7 subjects).
Another way to state this is that they hand-selected the people so that they fit the team’s preconceptions about what Christians should be—but did not pay so much attention to what non-Christians were (more on this later). The hope is that all this early manipulation would not impinge upon the results or the general applicability of the findings. Hope is a wonderful thing and accomplishes much, so we should not disparage Harris and his brothers for relying on it. Scientists are people too.