Received this email from a reader:
I took on board all I read on your website, and it has created confusion in my mind. I have been reading Ioannidis and others about the disagreement between observational studies and RCT’s. I know that it is impossible to adjust statistically for all confounders, as many are unknown in observational research. You pointed it out, and elsewhere too there is a lot of talk about the failings of epidemiology and its servants, the cohort and case/control study. I am about to enroll in a PhD where I will be conducting a observational study. Is observational studies and epidemiology a pseudo science? Should I enroll? Would it not be better to focus my energy on learning about clinical trials? You are the only person I know that can help me, it is driving me nuts! I do not want to pretend I can see the emperors new clothes…
Hope you can help me.
About whether to enroll, I am the worst guide. If pressed, I’d probably recommend a series of high risk Nigerian securities as the surest path to riches. I do know that running a blog is not a wise career choice.
But about the difference between observational versus controlled trials, I am your man.
There is nothing in the world wrong with observational studies. Indeed, nearly all the knowledge we develop is based on observation. Think of everything you’re certain and uncertain about. What is your wife/mother likely to say when she walks in the door at night? You don’t know for sure, but based on long observation you can make a good bet. You’re adept at guessing whether it’ll rain that day by looking at the morning sky.
Yes, you’re not using formal, i.e. cookbook, models, but what of it? Most probability, as I often say, is not quantifiable. Informal judgments are adequate for the bulk of life. I needn’t go on: you get the idea.
Now, assuming no calculation mistakes, all surveys, observational studies, polls and so forth are valid for the type or kind of data they represent. Thus (assuming no cheating, too) the polls flashed on the screens of Fox News and MSNBC are perfectly representative of the type and kind of people who would call or log in to those stations.
Which, given our daily observational life, tells us these people are not the same type and kind as other citizens. How applicable are the polls from the stations to the rest of the country? There’s no real formal way to know, but you can guess pretty well. Or you think you can.
Your giant observational trial, whatever it is, assuming no cheating, purposeful misinterpretations (e.g. the epidemiologist fallacy), and the like will also be valid for the kind and type of data it represents.
In it, you will be interested in some thing, call it Y. The goal of this study is to say, of the rest of the data, assuming the model is good and given that X1 = a, X2 = b, Xp = p, the probability Y = y is this-and-such. Simple as that.
The grand mistakes, and the reasons observational studies have poor reputations, are two. One is that people can’t stop themselves from making causal interpretations of the Xi. It may be in your data that as Xi varies over its range the probability Y = y changes a lot. But that does not imply that Xi is of any causal relation to Y.
Two is not knowing how the Xis came to be what they were: we took them as they came, just like in the TV polls. This mistake boils down to the same as over-interpreting the TV polls, claiming they say things about all Ys and not just the kind who would show up to your dataset.
Another difficulty, incidentally, is that people forget probability models are epistemological not metaphysical.
Experimental trials are not too different. Except that you in advance manipulate some of the Xi, accept the other Xj as they come just like in an observational study, and watch what the Xs do to Y. Experimental trials are thus always partly observational.
The reason experimental trials have a better and deserved reputation is that, over a long period of time, for the Y of interest, people have whittled down the Xi to a set that experience has shown are more closely related to Y, and some of which may even be causally related to Y (we still accept some Xj observationally). And they’re able to pick the Xi from the type and kind of area they claim the study represents.
Gist: there is nothing fundamentally wrong with conducting observational studies.