Somebody some enterprising young academic, not realizing he should keep his mouth shut before attaining tenure, will publish a study which examines the bizarraries found in studies which begin with the words, “We recruited participants online.” (See this one, for example.)
The peer-reviewed study by Shenhav, Rand, and Greene begins with the words “We recruited participants online.” They call it “Divine Intuition: Cognitive Style Influences Belief in God”, and it is found in the Journal of Experimental Psychology: General (sorry, no link).
Before revealing the full purpose of our trio, let me explain their three experiments. Bear with me through these: not everything is easy.
Via the internet, they recruited 882 folks, two-thirds female, and asked them if they believed in God or not-God. They also asked about “belief in an immortal soul, familial religiosity during childhood, and change in belief in God since childhood,” etc. They finally asked three “math problems with intuitively attractive
but incorrect answers.” They only tell us one, which went something like this (try to answer before reading the solution):
A bat and a ball costs $137.50 in total. The bat costs $112.20 more than the ball. How much does the ball cost?
This is solved (this is from me, not them) by a “system of equations”, the first of which is “Bat + Ball = $137.50”, the second of which is “Bat – Ball = $112.20.” Therefore, “Bat = $137.50 – Ball” (from the first) and then (substituting into the second) “$137.50 – Ball – Ball = $112.20.” Thus, “Ball = $12.65” which makes “Bat = $12.65 + $112.20 = $124.85.” Checking shows $124.85 + $12.65 = $137.50.
Simple, once you get the hang of it. That is, once the method is taught to you. By a teacher. Who most likely resides in a school or university. Which is the place you’d be sitting when you learned these things. And what kind of students and teachers are more likely to be better at questions like this? Well, math and science students, naturally.
What was your answer before you read the solution? Did you find an intuitively attractive answer? No? Let’s return to where I said the problem “went something like.” The problem actually went:
A bat and a ball costs $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?
Our trio says, “The response $0.10 springs immediately to mind, but the correct answer is $0.05. Choosing the attractive but incorrect answer signals greater reliance on intuition and less reliance on reflection.”
We math teachers call these kind of questions “gotchas.” Some less scrupulous pedants use them to show students that the students know less than the teacher. The question is a set up, designed as our trio said, to offer an answer which does not require much thinking. Student thoughts usually run along the line, “Who wants to think about a meaningless question about absurdly priced bats and balls? The bat’s a buck more? Gotta be ten cents.”
But some folks, science and math denizens, will recognize the “gotcha”, and work through the math. And so will the other people if the question is not presented in “gotcha” form, like I displayed originally.
So what we have here in this question is a reasonable filter to separate the math and science habituées from those folks who were once humanities majors. Used in this way, as a filter, the math problem would be unproblematic, but our trio did not use it as a filter.
Before telling you how they used it, let me ask you this question. Answer honestly. Among college graduates, who are more likely to be atheists, math/science or humanities majors? The former, of course. And at least some of this difference in attitude is due to acculturation and not deep and lasting philosophical inquiry. Christians (and I don’t mean “creationists”), for example, aren’t particularly welcome in biological circles, especially on the internet. On average. Not always. I mean, there is a tendency for acculturation to explain some but not all of the difference between theism and atheism. Nothing can be plainer than this.
Our trio took, for each participant a total of the wrong answers from the three “gotcha” questions. So everybody would have a score 0, 1, 2, or 3, with higher being “worse.” They then “correlated” this score with the answer the participants gave about belief in God. These two measures were linearly correlated (the wrong measure because of the discreteness of the score) to the tune of 0.18 (a bare whistle). The use of linear correlation can exaggerate this number, but at least this wrong one was accompanied by a wee p-value, which is a measure of success in academic studies.
The interpretation is that among those participants who believed in God, slightly, fractionally more of them scored poorly on the “gotchas” than those who did not believe in God. This was strange because the correlation between “Belief in God” and the separate question, “Convinced of God’s existence” was only 0.62, where one would have guessed it would have been unity.
Could it be an internet-recruited study pool just didn’t take the subject seriously? Well, our trio said they used a series of seriousness checks and tossed data that did not conform. So we are left with a puzzle. Or bad data.
Our trio’s explanation does not include the possibility that the slightly lower scores of the theists are because more atheists are found among those with backgrounds in math and science and of those more highly educated in general. Again, acculturation does explain some (not all) of the differences in belief. The authors summarily failed to recognize this. The effect I have in mind is not large, but then again neither was the effect found by the authors.
Study 2 was so similar to the first study that I refer interested readers to the paper. Relevant correlation here was 0.14, even smaller and well within the realm of explanation I proffer.
They had on-line volunteers write about themselves, with instructions such as:
“Please write a paragraph (approximately 8–10 sentences) describing a time your intuition/first instinct led you in the right direction and resulted in a good outcome.” Participants were excluded if they failed to write at least eight sentences.
The condition “your intuition/first instinct” (intuition) was half the time switched with “reasoning through a situation” (reflective), and the “right…good” switched with “wrong…bad” in a 2×2 design.
Sadly for the authors, a low p-value was not to be found in correlating of these four conditions (intuition/good, intuition/bad, reflective/good, reflective/bad) with belief in God. But they were able to do some data churning and find a publishable p-value in the “crossover interaction between the recollected cognitive approach and the valence of the recollected outcome.”
Still, this experiment, since it did not find the main effect, must be considered a bust.
This section of papers is where all the fun is had. It is where speculation is wild and typically free from the burdens of evidence. The same is true in press reports of papers.
Science magazine summarized the paper thusly, “People who chose more intuitive answers on these questions were more likely to report stronger religious beliefs, even when the researchers controlled for IQ, education, political leanings, and other factors.” Which sounds a lot stronger than what we have seen is true. But you can’t blame Science (this time) because our trio open the conclusion with the words, “[we] showed that intuitive thinking predicts belief in God.”
Enter the speculation (references removed):
The observed relationship between reliance on intuition and belief in God may stem from multiple sources. First, as noted earlier, belief in God may be intuitive for reasons related to more general features of human cognition that give rise to tendencies toward dualism, anthropomorphism, and promiscuous teleology.
Good grief! Promiscuous teleology? Who wants to suffer from that? The good news is that there is a cure and belief “can be overridden through the engagement of controlled or reflective processes, with reflective processes enabling or supporting judgments based on less intuitive explanations.”
Our trio then explain that people who believe in God are more likely to find evidence which supports this belief. No word on whether those who believe in not-God are more likely to find evidence which support their belief. But we are warned that “the belief in God may give rise to a feedback cycle whereby satisfying explanatory appeals to God reinforce the intuitive cognitive style that originally favored the belief in God.” No feedback cycle mentioned for atheists.
The paper ends with what you can see the authors hope is a literary zinger:
How people think—or fail to think—about the prices of bats and balls is reflected in their thinking, and ultimately their convictions, about the metaphysical order of the universe.
Now since there is not one word about the limitations inherent in recruiting people via the internet, and there is nothing but mute silence on the data discrepancies (noted above), nor is there even a hint or even the shadow of one about plausible alternate explanations (such as I gave), and since the third experiment which should have had the strongest effect if the authors’ theory was true but which was not a success, and because the authors offer a one-sided conclusion, let me end with this.
How researchers think—or fail to think—about the role of experimental design is reflected in their writing, and ultimately their convictions, about how to draw unjustified conclusions from weak data.
Update From Stephen Dawson comes pointers to the two other “gotcha” questions; and from that same blog (Jospeph Hertzlinger’s) comes more revelations (here and here) about an experiment that your adjunct did not have time to cover (yet).
See also this report.