This is going to start slow, unwind at a leisurely pace, and finish, for some, with a depressing conclusion, though we won’t get there today.
Here’s the setup. You’ve invented a new treatment to cure cancer of the albondigas and want to compare it to a placebo. Forget the ethics: this is a gedanken experiment.
Two groups of people: one gets your treatment, the other Obamacare. Kidding! The second is given placebos. Since this experiment is taking place in our imaginations, suppose there are no differences in the two groups, except one: whether or not a person has the treatment or a placebo.
Don’t gloss over no differences, because this is the key. It means what it says: no differences. The people are therefore identical in every possible way. They are all men (say), of the same age, born in the same town of the same parents and have eaten the same food down to the last morsel over their perfectly interchangeable lives. Their burden of disease is the same.
They have all said and thought the same things and have had the same things said to each of them at the same moments. They have breathed the same air, read the same books, drank the same whiskey. They are all named Bob. In short, there is nothing to differentiate one Bob from another. As in nothing. Get it? Nothing.
Half the Bobs get the treatment and half the placebo. While they’re getting their meds, their lives are as identical as before. The pills are popped simultaneously, administered by the same nurse while sitting in the same doctor’s office. With a proviso, no differences but the form of treatment up to and including the specific time and date (fixed in advance) which they are measured for a cure. There is no chance of measurement error.
Only one of four things can happen. (1) Every Bob is cured, (2) no Bob is cured, (3) every treatment Bob is cured and no placebo Bobs are, (4) no treatment Bob is cured and every placebo Bob is. It’s always all or nothing because the patients are identical down to the quark, or string, or whatever is at base. It cannot be that only some treatment Bobs are cured while other treatment Bobs languish because we assumed that there was no difference in treatment Bobs, not only before they received their meds, but all the way up to the time they were measured for a cure. Thus if the treatment “works” it must do so on all treatment Bobs or none of them. The same goes for placebo Bobs. The subject of this essay is to describe what “works” means and the evidence we have for it.
Outcome 1. What cured the Bobs? It can’t have been the treatment per se, because all Bobs got better. Was it the power of their minds via the placebo effect, since the treatment was as if it were a placebo? Or were they getting better anyway and would have been cured regardless of the experiment? We don’t and cannot know. Both hypotheses are possible.
The proviso hinted at above is now important. The treatment, since it is not the placebo, must have caused some difference between the Bobs, unless the Bobs’ bodies handled the chemicals in the treatment in precisely the same way as the placebo. This is not an impossibility. We can’t know if it’s likely, though, not with the evidence we have.
Perhaps the treatment hastened the cure in the Bobs that got it. We’d never know because we measured all Bobs for a cure at the same time. Or maybe the treatment impeded the cure, but not so much that it was noticeable. Point is, we cannot say anything about the treatment, for good or bad; not with the evidence we have.
Outcome 2. We can say with certainty that the treatment cannot cure cancer of the albondigas in the given time frame for people who are exactly like the Bobs. But that’s not an especially important thing to say. Why? Because how many people (beside the Bobs) are just like the Bobs? None. The treatment might cure not-Bobs.
Don’t be trapped thinking that the “degree” not-Bobs are different from Bob increases their chance the treatment might affect a cure in them. For one, there is no practical way to measure all the differences between Bobs and not-Bobs; in practice we’ll always have to choose some finite number of measurable attributes and can only speak of the differences in these. That none of these are different between some Bob and some not-Bob does not mean there are no other important differences. Second reason: we have no evidence, except hope, that the treatment is of any use.
We still can’t say what changes the treatment might have caused, assuming that the Bobs processed the treatment differently than the placebo. A cure might be forthcoming at some point after the official measurement period for just the treatment Bobs or just the placebo Bobs. Or a cure might forever be blocked in the treatment but not placebo Bobs, or vice versa. Or the treatment might have worked: every albondigas of every treatment Bob might have gone into remission at some point before measurement day, only to re-cancerify once more. We can’t know; not with the evidence we have.
And these are not the only scenarios for the two outcomes. Many more can be thought of, all equally evidence free.