Cornell’s Daryl Bem case is instructive. He’s an academic who has published several notable peer-reviewed articles which claim that (several different versions of) ESP is real. Trouble is, despite the prominence of the journals, and the peer review, almost none of his peers believe his results.
They publish his papers anyway because the papers meet the statistical criterion of success, which is to say the papers contain wee p-values, which are p-values less than the magic number. Bem always finds, at least in the papers he submits for publication, publishable p-values. In his latest work he touts, “all but one of the experiments yielded statistically significant results.” This is code for “p-values less than the magic number.”
This sets up a conflict in the mind of the researcher. Small p-values are thought to be the proof definitive. Yet it is clearly absurd, or at least extraordinarily unlikely, that people can read minds through time and over vast distance, or that they can, by grimacing and grunting, bend spoons using only the power of thought.
The obvious answer—ignore the small p-values and substitute for them a stronger form of evidence—never occurs to the skeptical researcher. Well, it couldn’t really because the researcher has never been taught any other form of statistics. At that is the fault of people like me.
But what the researcher can do, and does, is to question Bem’s experimental protocols. He picks these protocols apart. He shows how other, non-paranormal explanations are just as, or even more, likely to have caused the results. He shows where “sensory leakage” could have crept in and masked itself as extrasensory perception.
In short, disbelieving Bem’s theory behind his statistics, the skeptical researcher picks Bem’s experiments apart. Or skeptics just ignore the statistics knowing that some other explanation besides the paranormal must exist. And all this is good.
Put it another way. The researcher reading Bem’s papers acts as a scientist should, asking himself, “What else could have caused these results?” There must be an end to this question, of course, for it is always possible that an infinite number of things could have caused a certain set of results. But there will, given the evidence available, be a finite list of plausible causes which should receive scrutiny—and which are preferable explanations over ESP.
Now wouldn’t it be nice if researchers in these “softer” fields did this routinely? Not just for extraordinary claims like Bem’s, but for all claims, especially those preposterous claims (we’re sick of these examples, I know) like exposure to the American flag turns one into a Republican, exposure to 4th of July parade turns one into a Republican, or that fMRIs can tell the difference between Christian and non-Christian brains.
These absurd hypotheses never receive the scrutiny Bem endures not because the claims are any more likely, but because they are more likely to match the political and emotional biases of researchers. About the fMRI they might think: Christians are different from us, aren’t they? They at least believe different things. Therefore, their brains must be wired differently, such that the poor souls were forced into believing what they do. Besides, just look at those small p-values! The results must be true.
So today a toast to alternate explanations. May they always be sought.