The Gold Standard Of Experimentation Exposed As Pyrite

Everybody says the “gold standard” of experimentation, and especially clinical trials, is the randomized controlled trial. A typical view is that “A Randomized Controlled Trial is an experiment or study conducted in such a way that as many sources of bias as possible are removed from the process.”

This view is false. Randomization cannot remove bias, but it can add it. Even stronger, everybody, including proponents of randomization, has always known this.

You have handfuls of two seed varieties and a large field in front of you. Which seed provides greater yield? Well, plant the seeds, grow the crops, and then count or weigh the harvest. For example, on the north end of the field, plant seed A; on the southern, plant B.

But wait! Upon inspection, the field has a definite slope from north to south, and since water flows downhill, and water almost surely will affect seed growth, we should control our plantings so that equal numbers of A and B seeds are uphill and downhill.

Maybe those trees on the east side of the plot will cast too much shade before noon, so we can’t just put all A (or B) on the east side, and all B (or A) on the west. Instead, we can chop the field into four blocks: a northwest, northeast, southwest, and southeast. We then apportion the seeds A and B into these blocks so that each variety overall receives equal shade and water.

Is that it? Well, it turns out that soil analysis reveals that the nitrogen content across the field has a definite pattern. The plants will make use of this nitrogen, so we cannot ignore the unequal distribution of it. So, even though it’s difficult, we can chop the field into blocks such that water drainage, sunlight, and nitrogen are overall equal for both A and B.

Obviously, we can keep on doing this for each factor that we believe has a causative relationship with seed growth. Further, everybody agrees that this is a sensible strategy.

And so it is. Controlling for known or likely causative factors is just the thing. Experimenters in all fields practice control in their setups routinely. Chemists have standard atmospheres, physicists are careful to define all the conditions in which an experiment takes place, and so on.

So why is it that some would go further and ask for “randomization”? What does that do for us? Well, nothing. For example, after chopping our fields into blocks such that we are sure A and B are equally represented across environmental conditions, some would ask that A and B be “randomized” to these blocks.

That is, a (computerized) coin flip would decide the precise order of planting. Supporters of randomization say that this eliminates unseen bias. The field has been blocked, but there still might remain causative agents or factors that we did not measure that will affect seed growth. They say randomization assures us that these unknown, or unmeasured or unmeasurable, factors are evenly split between the seeds.

Mathematically, this is false. Since the number of possible causative factors that might affect seed growth are practically limitless—you’d have to specify the movement of every quark, or superstring, every photon, every graviton, etc. of the seeds, soil, sunlight and so on to be certain of equality—the probability that you have achieved balance between them all by a coin flip is as close to zero as you like. In other words, imbalance is guaranteed. (See this explanation for the math.)

It is control which is important, not randomization. Control is the true gold standard. Another example: suppose you commission a poll to ask, “How’s President Obama doing?” and then called only people in San Francisco. Would that sample be representative of voters across the States? Obviously not.

You want to control your sample for at least geography, if not income, sex, age, party affiliation, and so on. A collection of randomly dialed telephone numbers would be more biased than a carefully controlled sample.

Randomization can provide some benefit, but only in one narrow case: when the experimenter is likely to cheat. For example, in drug trials nobody trusts the pharmaceutical company, nor the doctors on its payroll, to choose who gets their drug and who the placebo.

The temptation for the drug firm to gives its pill to sicker patients, and the placebo to healthier, is akin to putting a bowl of candy in front of a hungry kid and saying, “I’m going out. Don’t eat any.”

True control—taking into account each patient’s history, genetics, etc.—by an outside disinterested agency would be better, but, baring that, a coin flip will do.