Have doubts about the latest finding from researchers? Confused that last week red wine “increased the risk of heart disease” but this week red wine “decreased the risk of heart disease.” Concerned about that the apocalyptic findings of environmentalist scientists might be mistaken?
You needn’t be. Because science—if you will allow the reification—engages in the act of replication (behind closed doors, and using white lab coats for protection).
In a recent Skeptical Inquirer, James Alcock was saddened that the Journal of Personality and Social Psychology, the journal which published Daryl Bem’s positive “finding” of pre-cognition, would not publish his (Alcock’s) replication of Bem’s experiment, this time with the conclusion of no effect, on the grounds that they (the journal) only publishes on subjects positively.
Alcock lamented the journal’s attitude, and claimed that they were violating the sacred tenet of replication. Even worse, the policy of letting in positive findings and the barring of all negative findings guaranteed over-certainty. Dr Alcock was right. But more interesting is why he was so.
It makes sense that if you only publish news of positive “findings,” and never let fall a whisper of negative ones, you are bound to occasionally cry wolf or to celebrate erroneously. But what does replication have to do with it? Why replicate at all? I mean, why do scientists themselves say that replication is sacred?
Obviously, and trivially, it is because they acknowledge that mistakes are not only made in experiments, but they are made often. Replication is the hope that the other guy won’t make the same mistake the first guy did. The word is “hope”, because it can and has happened that the other guy does make the same mistake, or makes new mistakes, or makes both new and old ones.
It is only human nature to feel that it is the other guy is the one more likely to make mistakes. This is why many replications are made by a second guy who is convinced the first guy is nuts. He feels that not only will the replication prove this nuttiness, but it will elevate himself as a beacon of Objectivity in the process. Two birds, etc.
Mistakes are more common when scientists are convinced, or nearly so, about the outcome of an experiment before the experiment is carried out. So intent are they in looking for the answer they know will come, they often find it, even when it isn’t there. This is why we find the, to some, curious phenomenon that effect size (for some experiment) decreases the more it is replicated.1
From this principle, we have the corollary that the second guy replicating the first guy’s alleged nutty work may himself fall into error, confusing his animosity for his enemy for the unlikelihood of his enemy’s results. And we mustn’t forget the possibility that both guys are wrong.
Another common time for mistakes is when scientists are in the grip of Theory. Not your everyday, mundane theory; no, sir. The Theories I mean must be bright, sparkling, beautiful, and large, in the sense that they purport to explain the Human Condition And How To Fix It. Results contingent on the Theory rise like a tsunami, an unstoppable force, washing away everything before them. Even proof of the Theory’s falsehood, via replication or logic, cannot hold the waters back. We must instead stand aside and wait for the force to spend itself.
The less objective or the more complex the subject, the easier it is to see your own reflection in the data. Anything to do with people is maddeningly complex and rarely wholly objective. This is why mistakes are made in these areas more than others. Physicists trying to discover which mixtures of atoms leads to the highest temperature superconductor are less apt to trip than sociologists on the hunt for “disparities and their causes.”
It is also so that the more encompassing the Theory, the clearer the levers of social control within it, the closer it aligns to one’s Way Of Life, the more beautiful it is, thus the more compelling it is and the easier it is to cherish.
But love is blind and lovers cannot see
The pretty follies that themselves commit;
Replication or not!
1Even though all the experiments had small p-values!