Academic Pscyhologists Over-Rely On WEIRD People: Overconfidence Results

It was just over a month ago that tears of joy came to my eyes while reading an academic paper. This was Simmons et alia‘s “False-Positive Psychology : Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.” (Thank you P-value!)

Right on the heels of that essential work we have Joseph Henrich, Steven J. Heine, and Ara Norenzayan’s lovely review, “The Weirdest People in the World” (58-page pdf) which asks two important questions: (1) “How representative are experimental findings from American university students?” and (2) “What do we really know about human psychology?”

Over-confidence and over-certainty abounds in many areas, the closer the subject is to mankind the greater the false surety. This is why this new paper is such a delight: it is a rare admission that all might not be as solid as hoped.

WEIRD = Western, Educated, Industrialized, Rich, and Democratic. That is, the kind of folks who make up 96% of the study participants “of the top journals in six sub‐disciplines of Psychology from 2003‐2007.” I.e. mostly young college students, and mostly American ones at that: about two-thirds of study participants are from the States.

“Researchers—often implicitly—assume that either there is little variation across human populations, or that these ‘standard subjects’ are as representative of the species as any other.” But it isn’t so. Our trio point out that

there is substantial variability in experimental results across populations and that standard subjects are particularly unusual compared with the rest of the species…The comparative findings suggest that members of [Weird] societies, including young children, are among the least representative populations one could find for generalizing about humans.

Conclusion?

[W]e need to be less cavalier in addressing questions of human nature on the basis of data drawn from this particularly thin, and rather unusual, slice of humanity…

In top journals such as Nature and Science researchers frequently extend their findings from undergraduates to the species—often declaring this generalization in their titles. These contributions typically lack even a cautionary footnote about these inferential extensions.

Amen, brothers; amen.

The authors looked at several different kinds of human behavior where there existed cross-cultural studies. For example, visual perception. We have all seen the Mueller‐Lyer illusion, which shows pairs of lines with arrows pointing inwards and outwards, as in this picture:

To many people the line segment in (a) appears shorter than in (b), though both are the same length. You can ask a person to lengthen (a) until it appears to be the same as (b). Weird people—on average—lengthen (a) about 13% (see Fig. 2; here reproduced). But peoples from some other cultures only need 1%; while still more need as much as 20 to 21%. This is clear and enormous variability across peoples, variability which is not accounted for in the vast majority of psychology journal statistical analyses.

Another area, heard about all the time in both evolutionary psychology and (so-called) experimental economics is how people think about moola. A favorite is the Ultimatum Game, in which one person of a pair is given some real money.

One of the pair—the proposer—can offer a portion of this sum to a second subject, the responder. Responders must decide whether to accept or reject the offer. If a responder accepts, she gets the amount of the offer and the proposer takes the remainder; if she rejects both players get zero. If subjects are motivated purely by self‐interest, responders should always accept any positive offer; knowing this, a self‐interested proposer should offer the smallest non‐zero amount. Among subjects from industrialized populations—mostly undergraduates from the U.S., Europe, and Asia—proposers typically offer an amount between 40% and 50% of the total, with a modal offer of usually 50% (Camerer 2003). Offers below about 30% are often rejected.

In other words, Weird people offer about 45% on average, but this hugely varies from between 25% to just over 50% across other cultures. Again, on average: the variability within a culture is also of interest and, though our trio don’t say so, must be different in different cultures too. Even worse for academics, Weird people are on the end of the distribution here (see their Fig. 3) making extrapolating from Weird results a dicey proposition.

The authors also survey folkbiological reasoning, spatial cognition, antisocial punishment and cooperation with anonymous others, independent and interdependent self-concepts, romantic love as a basis of marriage, moral reasoning, behavioral economics, several other areas, including one of especial interest to us: analytic versus non-analytic reasoning (see this post).

They find in each instance significant cross-cultural differences, with Weird people “frequently a distinct outlier vis‐à‐vis other global samples. It may represent the worst population on which to base our understanding of Homo sapiens.”

Importantly, they also recognize many cross-cultural similarities, such as perception of color, understanding of facial display of emotions, knowledge of numeracy, and theory of mind (i.e. all people recognize other human minds). Males everywhere rated female attractiveness important in mate selection. Nobody anywhere (except those at the Occupy rallies) likes free riders.

To fix the over-reliance on Weird folk, the authors list some recommendations, such as that “Journal editors and reviewers should press authors to both explicitly discuss and defend the generalizability of their findings. Claims and confidence regarding generalizability must scale with the strength of the empirical defense.” They also say to use people from more areas of the world.

Anybody care to wager on whether these recommendations will be broadly adopted?

The paper is endlessly quotable and I can do no more than summarize its main points. I urge all interested people to read the original report: there is much material to digest.

—————————————————————————

Thanks to Stephen Dawson who suggested this topic. See also Jospeph Hertzlinger’s blog on this topic.