Watch out Sam Harris, Gordon Hodson and Michael A. Busseri of Brock University are giving you competition for the worst use of statistics in an original paper.
Their “Bright Minds and Dark Attitudes: Lower Cognitive Ability Predicts Greater Prejudice Through Right-Wing Ideology and Low Intergroup Contact” published in Psychological Science1—headlined in the press as Low IQ & Conservative Beliefs Linked to Prejudice—is a textbook example of confused data, unrecognized bias, and ignorance of statistics.
Hodson and Busseri on are track to beat out Harris’s magnificent effort, and they might also triumph over the paper which “proved” brief exposure to the American flag turns one into a Republican and the peer-reviewed work “proving” exposure to 4th of July parade turns one into a Republican.
Let’s see how Hodson and Busseri put themselves into the running.
The authors intimate that “individuals with lower cognitive abilities may gravitate toward more socially conservative right-wing ideologies that maintain the status quo and provide psychological stability and a sense of order”. They say that this “is consistent with findings that less intelligent children come to endorse more socially conservative ideologies as adults”.
How did they prove that idiots and conservatives are racists? They gathered two large data sets from the UK, one started in 1958 (NCDS), the other in 1970 (BCS); about 16,000 individuals in total, roughly equal numbers of males and females. The quizzed the groups when they reached 11 and 10 years old on their “intelligence”; they then came back to these individuals when they were 33 and 30 and asked them about their “socially conservative ideology and racism.”
The authors do not say how many people they used in their analysis; how many individuals were lost in the 20 years between surveys is not noted in their paper. My read of the NCDS website (pdf) makes the loss about 30%. That leaves about 11,000.
Intelligence was defined in one database as scoring well on matching the similarity between 40 pairs of words, and on matching the similarity of between 40 pairs of shapes and symbols. On the other database, this changed to drawing 28 missing shapes, recalling digits from 34 number series, identifying the definitions of 37 words, and “generating words that are semantically consistent with presented words” 42 times.
Thus the two samples measure similar but different abilities. The NCDS (pdf) also had available the Peabody Individual Achievement Test Math and Reading sub-scales which were not used as intelligence measures. Why?
When the kids became 33 and 30 year olds, they were asked whether they agreed with 13 or 16 questions like, “Schools should teach children to obey authority”, “Family life suffers if mum is working full-time.”
Another was, “People who break the law should be rehabilitated.” Just kidding! It’s actually, “People who break the law should be given stiffer sentences.” The bias in the question wording is ignored.
Another question was, “None of the political parties would do anything to benefit me.” Is agreeing or disagreeing with that a “conservative” position? What would the Occupy people say? Another, “Being single provides more time to experience life and find out about yourself.” Conservative or liberal?
According to the NCDS (pdf), there were about 50 questions, of which only 13 were used. A “conservative”, then, is whatever Hodson and Busseri say it is. The same thing goes for what a “racist” is.
For these questions “reliabilities ranged from .63 to .68.” This means the questions are imprecise and imperfect, so that if you use the raw results in subsequent analysis, you must “carry forward” the uncertainty in reliability. Did Hodson and Busseri do this? No.
One would have guessed from the title, that the authors looked at how the scores on the intelligence questions correlated with the scores on the attitude and racism questions, taking into account the uncertainty in the reliability. You would be wrong.
They first modeled the intelligence questions to create one “latent” (unobserved) measure, called “g”. The uncertainty in creating “g” is then ignored in all subsequent analysis. They did the same for the attitude questions, creating a “latent” (actually unobserved) variable called “conservative ideology.” Uncertainty in its creation is also ignored. Then the individuals’ education and socioeconomic status and separately their parent’s socioeconomic status (which again were the results of models) were put into a model with “g” and “conservative ideology” to predict “racism” (the uncertainty of which, as was already said, was ignored). The picture below summarizes their findings.
Lo, they found small p-values. The authors appear unaware that samples of this size are practically guaranteed to spit out small p-values.
What makes the study ludicrous, even ignoring the biases, manipulations, and qualifications just outlined, by the authors’ own admission the direct effect size for “g” on “racism” is only -0.01 for men and 0.02 for women. Utterly trivial; close enough to no effect to be no effect, their results statistically “significant” only because of the massive sample size.
The effect size for “conservative ideology” directly predicting “racism” is higher (0.69 and 0.51). But all that means is that the questions the authors picked for these two attitudes are roughly correlated with one another. In other words, “None of the political parties would do anything to benefit me” is crudely correlated with “I
wouldn’t mind working with people from other races” and so forth.
Yet the authors have the temerity to conclude, “These results from large, nationally representative data sets
provide converging evidence that lower g in childhood predicts greater prejudice in adulthood and, furthermore, that socially conservative ideology mediates much of this effect.”
Truly, statistics can “prove” anything.
—————————————————————————
1doi:10.1177/0956797611421206
Thanks to reader Jonathan Woolley who suggested this study.
Update I saw, on one website which linked to my criticism, a criticism of my criticism (get it?): “The subjects in the test were given a fifty question questionnaire and only 13 questions are used, and this jackass is complaining about that?” I am the “jackass.”
This articulate person (language warning on the link) says that social scientists mix in red herring questions with “real” ones so that interviewees can’t figure out what’s going on. This person also says that I was unaware of this. Not true. But even if I was, it would have been irrelevant.
The point I made was we do not know how the questions the authors did use—it doesn’t matter how many others were rejected and why these were chosen—were used to create “conservative” and “racist” indexes. I have given examples of two questions which are at least ambiguous; there are more. “Conservative” and “racist” are defined as how the authors see them, and not necessarily how civilians and other scientists would see them.
See also my comments below: the models fit by the authors result in very small effects. These effects mostly have small p-values, but as I said above, small p-values are practically guaranteed in large samples (> 1000). And remember, none of the uncertainty in creating the latent “g” and other indexes are carried forward in their models: if if was, the effect sizes would decrease further (and p-values would increase).
And for the real kicker, if we then “integrated out” the parameters (the βs) and tried to predict whether a person with a low “g” would be “racist”—the reason given for the study—the effects would be lower still, probably negligible. The “direct effect” was already trivial, the “total effect” barely marginal.
Incidentally, if you don’t know, “latent” means unobservable (and uncheckable). Social scientists love using these kinds of models—structural equation models, factor analysis, etc.—because they are so fertile. Sprinkle a little data on them and publishable p-values a plenty will sprout instantly.
When you believe you are an intellectually and morally superior person, it’s not surprising your study confirms that.
I’m thinking of dong a study to determine if exposure to death turns you into a democrat. There’s considerable anecdoetal evidence that most dead people vote democrat.
Ray,
This is brilliant. If we can get data from some of these voting fraud cases (Chicago, Brooklyn, etc., etc., etc.) we could do this. If you’re really interested and have the time to track these down, let’s do it.
This study and others that have been discussed here consistently suffer from two critical flaws. Briggs correctly points out the shoddy and misleading use of statistics. There is a second and more fundamental issue that involves flaws in the design of the study. (In fairness Briggs does point these out.)
In today’s post the fundamental problems begin with confusion about what is being measured: liberal, conservative, intelligence and so on.
Screwing up the study design the authors then go on to further the screw up with some shoddy statistical sleight of hand.
So we might refine the (correct) Briggsian conclusion and say that “Truly, statistics coupled with a crummy study design can and will ‘prove’ anything.”
Thanks for bringing these papers to my attention, Matt. I’ve always suspected that everyone who has ever disagreed with me is an idiot. Now I know that it can be statistically proven. 😉
I got into an online argument with a Brit recently regarding a statement about Left and Right wing politics. Most of the argument was due to a completely different understanding of what the two things meant.
For him, Right wing views were all about stuff like hierarchy and keeping people in their place. Left wing was basically the opposite. This seems like a silly way to talk about politics to me, since you can find both of these all over the place on both the American Left and Right. I guess it maybe makes more sense in a place that still has hereditary aristocracy.
So, in addition to a confused study design and misapplication of statistics, we probably are also dealing with a British based definition of Right wing.
Matt, given that the current UK (conservative) prime minister probably has more socialist tendencies than President Obama, I think you are correct that “left” and “right” are very much a matter of perspective.
The good Dr Briggs has already pointed out the manipulation inherent in only using 13 of 40 questions and the bias of the authors is clear from the implicit linking of conservatism with racism. Of course, when such bias is in agreement with the a journal’s editors, no-one seems to be able to see it.
I had a look at the paper and agree that it is, at best, an extremely sloppy piece of work. Briggs: Have you ever considered submitting a commentary or rebuttal paper to journals that print such dreck? Most have a paper category specifically for this purpose. Going that route would guarantee that the authors would see your critique, and would create a lot of pressure for them to respond.
What is the p-value in parentheses between “g” and “Generalized Racism”?
Is it not notable that that direct p-value is near zero when neither of the p-values on the other path (g->CI, CI->GR) are near zero? What correlations are being left out?
Sorry, not p-value, path coefficients.
I may just be misinterpreting what you’re saying here, but this was not my reading of the paper. The -0.01 and 0.02 were for path c’, which was after they accounted for the effect of “right-wing ideologies” on the relationship between general intelligence g and prejudice. As the paper states:
Which is what they allege to have found with the nonsignificant (p>0.05) -0.01 and 0.02 values.
Eli B,
Good point; I changed my text to remove the ambiguity. In their Table 2 the direct effect for “g” was “non significant” and trivial (but “significant” in the BCS for women). The total effect was “significant” (with asterisks!) but also trivial. The correlation is also small, and useless to us because they never define what “conservatism” or “racism” is. But even if the had, the correlation with, say, “g” is small. And the correlation, as near as I can tell, does not adjust for the socio-economic status, i.e. education.
Then let’s don’t forget that p-values always overestimate the confidence we have in any future observations (by “always” I mean “always”). Even if the rest of the errors I mentioned were ignored, it would still be the case that the confidence we should have in new “dumb” (low ‘g’) people being “racist” is small.
And all this is predicated on us accepting the model, which I don’t, not for a second. Such a convoluted thing!
What a stupid definition of conservatism was used for this study. Reminds me of the saying: ‘The god that you don’t believe in, I don’t believe in either’.
Make up a list of troglodyte beliefs, define that as ‘conservatism’ and then see who agrees with that list. Brilliant.
Then wait for the media to unquestioningly publicize the ridiculous ‘study’.
Now, who’s stupid??
P.S. By equating authoritarianism with conservatism, the study authors neatly turned the core of conservatism (weak central government, emphasis on state’s rights and individual freedoms) on its head.
I’ll admit to my stats being somewhat rusty, and not entirely following all the points you outline above, but it’s clear that you have a number of issues with the methodology of the study.
My question therefore is, how would you improve on their methodology, and if you were to design a study to study something similar, how would you design it to get the best result possible?
Pingback: Monbiot and The Intellectual Inferiority of Those With Right Wing Views | Libertarian View
The conflation (awful word “conflation”) of the right wing with bigotry was enough for me to move towards dismissal. Can left wingers not be bigots? Heard of the BNP? We shouldn’t be surprised by scientists using their research to push their own socio-political agenda, but we should be appalled; whatever their political hue.
Pingback: How to use bad stats to justify calling conservatives stupid… « mixal's cratch pad
Matt:
Speaking as a right wing Brit I think you’re analysis of what constitutes left and right wing here is a little narrow and I don’t think aristocracy comes into it at all.
My take is that left wing here means: redistributive tax polices, ‘progressive’ politics where the state intervenes more often to make things ‘fair’ with a focus on rights. So for example you have polices like support for quotas for women and minorities, higher taxes for the rich and more generous welfare.
The right wing here means: economically liberal, less state intervention in general and more trust in the market leading to lower state spending and lower taxes with a focus less on rights and more on responsibilities.
‘Conservative’ as meaning being against change isn’t really relevant. More Conservative party votes are in favour of massive change (by leaving the EU) than their Labour voting contemporaries.
Having said all that I agree that what counts for left and right can be cultural and the UK position does not necessarily translate to the US. For example gun control is neither a left or right issue here with the vast majority in favour.
I dismissed this study on a priori grounds
http://dissectleft.blogspot.com.au/2012/01/uninsightful-look-at-racist-attitudes.html
so did not bother to look at the actual stats — but they are truly pathetic, even for psychologists
Psychologists also have very peculiar ideas about what constitutes conservatism
Their measures of it often do not correlate with vote at all
JR
A more extensive commentary:
http://www.bloggernews.net/127686
JR
You have a link from the Telegraph:
“Stop Press: Gordon Hodson and Michael A Busseri’s study has been described as “a contender for the worst use of statistics in an original paper ever” by Dr William M Briggs, Professor of Statistical science at Cornell (hat tip: Libertarian View).”
http://blogs.telegraph.co.uk/news/tobyyoung/100135439/george-monbiots-worst-ever-guardian-column-%E2%80%93-and-thats-saying-something/
I feel Ruairidh may be slightly disingenuous. Traditional British right wing values are
monarchy, empire, country and respect for authority. Liberal, neo-classical, economics is a rather recent addition. It is certainly at odds with the rigid imperial trading system of which the right, including Churchill for example, were so proud. It would not be fair however to construe these values as (any longer) supporting a class hierarchy.
Pingback: The Progressive Mind » Monbiot and The Intellectual Inferiority of Those With Right Wing Views | Libertarian View
Pingback: Stop The ACLU
British perspectives tend to assimilate political positions far more into a simplistic left/right divide than in the USA. I give as an example, myself. I was able to obtain two degrees in philosophy from a UK university without being aware that a slicing of political views into authoritarian/libertarian segments is just as valid. When, now, I present people with David Nolan’s Diamond Quiz (for example here: http://friesian.com/quiz.htm), it is a marvel to behold the resulting illumination.
Pingback: The BBC: the world's largest liberal echo chamber – Telegraph Blogs
Winston:
I was not being disingenuous. The position I described is exactly how I see it today (admittedly as one individual). I think you may be describing a kind of pre 1980’s view of the British right that I don’t really recognise and have never known.
I am in my thirties so am not old enough to remember the British right in the time of an empire and don’t think many people here would use imperial standards as a basis of their politics even if they can remember them. The only people who use the empire & monarchy as any kind of political touchstone are those on the British left who want to set up and attack straw men caricatures.
I would agree that country and respect for authority are ‘traditional’ values of the right but I think many on the left would claim these too; they are not very controversial.
Briggs:
I think you need to review an undergraduate statistics text on the basics of mediation as you don’t seem to understand the concept. Furthermore, studies involving correlation don’t “prove” anything but rather suggest a relationship or association between variables. You can’t infer causation from correlation. These are standard concepts in most undergraduate statistics courses. I’m confused as to how you could make such naive errors considering your self-proclaimed status.
Aiken
I can understand how you can become easily confused, Aiken. Show me where I have, and the authors have not, confused correlation with causation. Show me exactly where I have misunderstood (the dicey, often misleading, linear-assuming concept of) mediation.
Just to confuse matters even more “Conservative” and “conservative” have two separate meanings in the UK; the capital denotes the centre-right wing party, conservative having the same meaning as in the US. So you can be a Conservative and be a million miles from being a conservative (Margaret Thatcher for instance).
As for the 30% drop out rate, I’m sorry but I don’t believe people were successfully followed for 20 years AND filled out the forms for no reward. It doesn’t ring true to me, but then I’m not a conservative. Maybe those filling in the forms and returning them were more likely to be conservatives?
Finally there are members of the Conservative led govt that would definitely be considered left wing lunatics in the USA – Ken Clarke for instance.
Well as Dr. Briggs says the paper just set out to support the pre-conceived position of the researchers. They would probably describe themselves as Fabians.
Those who don’t know what a Fabian is can find a reasonable definition on line. Basically they are middle-class people who see themselves as the saviour of the working class. To paraphrase an American commentator from the past – a Fabian is someone who thinks he owes a debt to the poorer citizen, and intends to dispense that debt with your money. They seem to believe that all lower paid people covet the money of the better paid (and should do of course).
The whole idea that you can work these attitudes out statistically (ignoring the dubious stats) is ludicrous. You have to start with a pre-conceived position. How do you define liberal/left – how you vote? Well that would give you as many, if not more, racists on the left in the UK. The British National Party (BNP) – seen as racist by many, draws more from the left than the right in elections. Just this aspect alone invalidates the outcome, without even inspecting the methodology.
Sorry to be political BUT If you want a perfect correlation then inspect the left in the UK against their time in government. Five times in power – 5 times left (pun intended) with the country in a worse financial state. Current mess and the 70’s IMF bail-out included. I don’t even need to apply student’s t-test for small samples!!!!!
I have not got much time for the right either – so I suppose that makes me a closet lefty then, especially as I don’t mind working with people of other races.
The whole thing is ridiculous.
So, Fabians = Liberal elite = the people who are destroying us.
Yet another link to this post from Tim Worstall:
http://www.forbes.com/sites/timworstall/2012/02/07/scientific-paper-conservatives-are-stoopid-so-there/
He discusses the confusion over what constitutes conservative or left wing in the study and more generally.
Pingback: Club Troppo » Missing Link Friday – Conservatism, prejudice and intelligence
https://www.wmbriggs.com/post/5118/
American conservatism is stupid by definition, no need for stats just listen to what they say you will realise that they contradict themselves over and over again. Conservatives like to call themselves christians i.e. followers of christ yet they hate socialism, anyone who has ever heard of the synoptic gospels knows that jesus was a socialists, other contradictions involves saying america is great and then saying “lets,make america great again”. Or sayin obama is week and then say he is an authoritarian
This is one of the worst supposed “take-downs” of a paper I’ve ever read. The number of ways in which you have mislead the reader are numerous.
“They gathered two large data sets from the UK”
– THEY didn’t gather anything. They were using data collected by others. This phrasing appears to create the impression of “bias” in the measures.
“The authors do not say how many people they used in their analysis; ”
-Sure they do. It’s right in that figure you posted. 4267 males, 4537 females.
“The bias in the question wording is ignored.”
This isn’t fair. Subjects can either agree or disagree with the statement on a scale. They’re not suggesting to subjects to agree to this. Subjects either agree or not, and the extent to which they agree determines where they fall on the conservative scale. (Usually these kinds of scales also have questions which are worded in the opposite, though I’m not familiar with this exact scale).
“A “conservative”, then, is whatever Hodson and Busseri say it is. The same thing goes for what a “racist” is.”
-From their paper: ” socially conservative ideology was assessed in terms of respect for and submission to authority (7 items in the NCDS and 10 items in the BCS; e.g., “Give law breakers stiffer sentences” and “Schools should teach children to obey authority”) and support for conventional (i.e., unequal) sex roles (6 items in both studies;” They’re being clear about what it is.
also for racism: “Attitudes toward racial out-groups were assessed in the NCDS and the BCS with the same five items (e.g., “I wouldn’t mind working with people from other races” and “I wouldn’t mind if a family of a different race moved next door”; ?s = .82; Deary et al., 2008; Schoon et al., 2010).” If you don’t like the methods compare it to the papers they’ve cited. But seems to me they’re following standard procedures here.
“One would have guessed from the title, that the authors looked at how the scores on the intelligence questions correlated with the scores on the attitude and racism questions, taking into account the uncertainty in the reliability. You would be wrong.”
-They did, more or less. The latent factor combines the intelligence measures, and the 0-order correlations with racism are shown and highly significant.
“Lo, they found small p-values. ”
-No they didn’t. On the path between g, conservatism, and racism, all ps < .001. That's not "small".
"by the authors’ own admission the direct effect size for “g” on “racism” is only -0.01 for men and 0.02 for women."
-I think you don't understand the stats here. They're saying the direct effect is small BECAUSE conservative ideology is the mediating variable between g and racism. (The 0 order correlation are -.19/-.17).
"The effect size for “conservative ideology” directly predicting “racism” is higher (0.69 and 0.51). But all that means is that the questions the authors picked for these two attitudes are roughly correlated with one another. In other words, “None of the political parties would do anything to benefit me” is crudely correlated with “I
wouldn’t mind working with people from other races” and so forth."
-And yet g predicts the conservative ideology. This is how the path works.
Again, I think you don't even understand the stats. Clearly, failure to understand stats can be used to "disprove" anything.
Pingback: IQ?????????????????????????????????????? - 2ch.LOGUE
Phineas Gage,
Clearly, an Adjunct Professor of Statistics at Cornell with a PhD in Statistics from Cornell does not understand statistics.
Sad that a professor of statistics who is disproving an opponents bias is clearly biased in his conclusion.
Anti-liberal propaganda.
Cornell might need a new statistics professor.
It’s not like this was an isolated study. There is tons of research that covered similar territory and corroborates the findings.
Is it impolite to discuss cognitive differences between liberals and conservatives?
by Gordon Hodson
https://www.cambridge.org/core/journals/behavioral-and-brain-sciences/article/is-it-impolite-to-discuss-cognitive-differences-between-liberals-and-conservatives/A88EB2E4FAEEA36231172A46EA467414
Low IQ & Conservative Beliefs Linked to Prejudice
By Stephanie Pappas
https://www.livescience.com/18132-intelligence-social-conservatism-racism.html
The Association of Cognitive Ability with Right–Wing Ideological Attitudes and Prejudice: A Meta–Analytic Review
by Emma Onraet, Alain Van Hiel, Kristof Dhont, and Gordon Hodson
https://www.researchgate.net/publication/282421724_The_Association_of_Cognitive_Ability_with_Right-Wing_Ideological_Attitudes_and_Prejudice_A_Meta-Analytic_Review
@Benjamin David Steele
Lol the second paper is just the same as this article. The first paper also came out after the article paper (2014 vs 2012). It even shares an author, Hodson.
The meta-analysis came out years later (and this paper is cited in its, one of the authors is even the same).
Weak correlations too for that matter, the percentage of variance explained is like, what, 4%?
@ Dude Its funny how you reject the very valid claims and criticisms of the study with your emotionally driven propaganda. It seems that Cornell is doing just fine with its statistics professor and it is actually you that needs a new brain. Your Anti-conservative attitude is shining bright, isn’t it?
@liberal Your comment does nothing to disprove the criticisms of Briggs, so I’m not sure why you bothered to waste your time posting this comment. Not all conservatives are religious and not all liberals are atheists. You can easily verify this by touching grass. Your definition of conservative is agreed upon by literally no one. But go on.