He gave the quiz’s score a super impressive name: “Science Comprehension,” as if top marks indicated you comprehended all of science instead of showing you answered correctly questions which wouldn’t stump the kids on Are You Smarter Than A Fifth Grader?
Kahan inflicted the quiz on a bunch of people and asked them things like whether they suffered from “religiosity”, had been awarded a “degree”, or were members of the Tea Party.
Lo, slightly more of those awarded a “degree”, a certificate indicating four or more years of expensive education in all things science, scored better than those without a degree. This shows that spending all that money doesn’t buy you much. (Alternate explanation: Kahan over-sampled “Studies” majors.)
Knowing whether a person suffered from “religiosity” meant nothing to knowing how a person would score on the quiz. This will chagrin those who hoped that belief if God precluded knowing that “Electrons are smaller than atoms.”
Perhaps worst of all, knowing whether a person was a Republican or Democrat, or whether a person joined the Tea Party or not, meant almost nothing to knowing how a person would score on the quiz. If anything, ever-so-slightly more Tea Party members knew that “the Earth [goes] around the sun” than non-Tea Party members.
Kahan was shocked by his discovery:
I’ve got to confess, though, I found this result surprising. As I pushed the button to run the analysis on my computer, I fully expected I’d be shown a modest negative correlation between identifying with the Tea Party and science comprehension.
But then again, I don’t know a single person who identifies with the Tea Party. All my impressions come from watching cable tv — & I don’t watch Fox News very often — and reading the “paper” (New York Times daily, plus a variety of politics-focused internet sites like Huffington Post & Politico).
I’m a little embarrassed, but mainly I’m just glad that I no longer hold this particular mistaken view.
This admission was manly—and extremely rare. But he (almost) spoiled it by running as quickly from it as he could, lest his colleagues think him insane, by immediately calling Tea Party members immoral brutes.
Strange that he came to that conclusion based on no direct observational evidence. Not very scientific behavior, that.
This isn’t Kahan’s first foray into “science comprehension.” He and several others were responsible for media-noted “The polarizing impact of science literacy and numeracy on perceived climate change risks.” (See notes below.)
Much the same thing: small trivia quiz matched with iffy political affiliation questions. Then an enormous and creaky statistical apparatus erected on top of these thin reeds with the purpose of teasing out infinitesimal differences. My God! Did they go on and on and on and then on some more? Did they ever. All completely worthless.
These kinds of “studies” come out with depressing regularity. By “these kinds” I mean those concocted by professors who have found ways to escape teaching by engaging in “research.” They gather in the nearest Starbucks (faculty lounges are passé), think of questions which they convince each other are profound and which support a fashionable theory, call the battery of questions an “instrument”, give it to a few students, then analyze, analyze, analyze.
Seriously, now. Just how far can you go with a question like “Lasers work by focusing sound waves [true/false]”? Could you tell how the country should best be run by gleaning insights from a survey which showed that only 69% of respondents knew the right answer? Could you tie the way a person answered that to some obscure psychological, jargon-laden theory? How plausible is it that most people’s politics and religious beliefs would cause people to answer correctly or incorrectly this particular question?
Why not require people to know what happens when a teaspoon of Seaborgium is mixed with a pinch of Livermorium as a prerequisite for marriage? How about insisting people know which has more mass, a neutrino or top quark, before allowing them to purchase soda pops larger than sixteen ounces?
Or what about ensuring a man can derive the fundamental equations of radiative transfer before he’s allowed to create a quiz which supposedly defines “science comprehension”?
It’s all so silly.
—————————————————————————————
Thanks to the many, many readers who alerted me to this study.
Incidentally, we met Dan Kahn before: Confirmation Bias Happens To The Other Guy, and Kahan’s Science Literacy Paper.
Good one Briggs, and how many think that you just made up those elements? If you every find enough to mix them, let me know. The problem here is that Dan suffers from lawism.
I really enjoy this site. I read a lot about Dan Kahn; I even took the NSF Science Literacy test and got a perfect score. I conclude: there is no relationship between scientific acuity, political affinity, and faith. Science is a process of discovering new knowledge. Politics is a process of deciding what a group of people can or cannot do. Faith is a belief. I don’t see the value of attempting to predict (poorly) how the knowledge of science informs on political viewpoint as modified by a faith prospective. Professors who teach well create value. Professors who discover how to mass produce carbon nanotubes through research create value. Professors who worry about the public’s perception of science and the political affinity and religious belief are lawyers. Lots of words, no value, and on reflection a lot of politicians are lawyers….
Scotian,
I was going to say that the question is wrong, the life-times of the two elements are so short that although you could add a pinch of Livermorium into a teaspoon of Seaborgium, you would wind up mixing a pinch of Flerovium into a teaspoon of a Rutherfordium/Seaborgium mixture.
“I don’t see the value of attempting to predict (poorly) how the knowledge of science informs on political viewpoint as modified by a faith prospective.”
That was the point. Dan is trying to stop people doing exactly that, by pointing out that there *is* no material relationship. A correlation r of 0.05 means an r^2 of 0.0025, which means about a quarter of 1% of the variance can be ‘explained’ by Tea Party membership. It’s utterly trivial.
Unfortunately, in trying to counter the expected reaction of liberals to finding there was such a small but borderline significant difference between liberals and conservatives, he said in effect that the Tea Party were better at science, which all the talking heads immediately picked up on. He was a bit annoyed that they got it so exactly backwards – his intended point was that they’re *not*.
There are distinctive differences between the parties in what bits of science they believe, but this is *not* because one party or another is better educated about science. It’s because that’s what they believe *politically*. Liberals believe in global warming because they’re *liberals*, not because they’re any better (or worse) at science. In fact, the more they know about science, the more quintessentially *liberal* in their beliefs about science they become.
Until you understand the problem, you have little chance of finding a solution. But whatever.
He’s just doing his job.
It’s the pseudo-science that rankles…
But this is well worth a read!
I meant this:
http://www.culturalcognition.net/blog/2013/10/19/congratulations-tea-party-members-you-are-just-as-vulnerable.html
Sorry. (I forgot the trick for making anchored links work from this blog’s comment section, and don’t have time to look it up right now.)
@Briggs: Still no editing tool for commenters? :sigh:
M. Jackson
Nice link, and thanks for it. A typo in it gave me a new word for certain types of social science studies, “comfortbabel”. The article in fact spends some time on how there is a strong tendency to overvalue comfortbabel which make one feel comfortable with one’s biases.
I like it! :)Thanks, max.
Maybe the test should have been on “numeracy” rather than science trivia.
At least the tea partiers are aware that budgeting more spending than you are taking in makes you wind up like Detroit, Stockton CA, Vallejo CA, or the Weimar Republic
Oh, boy! The Pauline Kael Fallacy in operation! A classic.
JJB
It’s even more bizarre than that. I tried to engage with Kahan on his web site. I kept getting distracted by a commenter making odd comments that almost made sense but didn’t. Here was the first comment from this strange commenter:
I clicked on the “I am dmk38” link … and came away more confused. I asked Dan Kahan what was going on with the strange commenter. Dan Kahan said it would all become clear after while, viz:
Imagine my surprise when, after more faffing about, I found out that dmk is a sock-puppet for Kahan himself! I was blown away.
I’ve never heard of such a thing, either before or since. The guy is operating a sock-puppet on his own website … can you say bizarre?
I wrote him off from that moment on. When someone finally took pity on me and told me that dmk38 was a sock-puppet for Dan Kahan, I wrote:
I’ve seen nothing since to change that view.
All the best, Matt, keep up the good work.
w.
Matt? Typing too fast. I meant William of Briggs of course.
w.
Kahan isn’t even the worst offender of this kind of stuff. That would be this guy:
http://woods.stanford.edu/about/woods-faculty/jon-krosnick
Strange bit of irony here, I actually met this guy when we were 11. And all I can say is that if you’re an insufferable 11 year old, you’re like to make an insufferable adult.
Have you been elevated Lord Briggs? Maybe this is the family line:
http://en.wikipedia.org/wiki/Asa_Briggs,_Baron_Briggs
We’re all sock-puppets here!