We discussed this before, but since it has come up recently in personal discussions, I wanted to offer this clarification.
Suppose we’re in a standard epidemiological situation, or even a planned experiment, where we have two groups, (a) exposed to some horrid thing, and (b) not exposed. It should be clear that the people in group (b) were not exposed. Where by “not” I mean “not”. We track the incidence of some dread malady, or maybe even maladies, in the two groups.
We collect data and “submit” it to some software, which spits out a wee p-value for whatever “test” you like between the groups; and where we can even grant—and upon this I insist—group (a) shows the higher rate of whatever dis-ease we track. It also shows a healthy predictive probability difference between the groups.
Lo, all statisticians would say the exposure and malady are “linked”, by which all of them would at least secretly mean “cause”. Whatever it was those in group (a) were exposed to caused the malady or maladies.
If you press them, and tell them they will be quoted and held accountable to their judgement, the statisticians may well lapse into “link”, and shy away from “cause”. But they will secretly believe cause.
Now here is what happened. Not everybody in group (a), the exposed group, will have developed the malady (or maladies; after which I use the singular to save typing), and some people in (b), the not exposed group, will have the malady.
It thus cannot be that the people in group (b) had their disease caused by the exposure. It then necessarily follows that their malady was cause by something other than the exposure. This is a proof that at least one more cause than the cause of the exposure exists. There is no uncertainty in this judgement. Not if it is true none of the people in the not-exposed group were not exposed.
Of course, it could be that every person in the not-exposed group had a different cause of their malady. All that we know for certain is that none of these causes were from the exposure.
Wild, right?
It’s worse, because even though we have proved beyond all doubt that there must exist a cause that was not the exposure, we have not proved that any people in the exposed group had their malady caused by the exposure. Why?
Because it could be that every person in the exposed group had their disease caused by whatever caused the disease in the not-exposed group—or there could even be new causes that did not affect anybody in the not-exposed group but that, somehow, caused disease in the exposed group.
It could be that exposure caused some disease, but there is no way to tell, without artificial and unproven assumptions, how many more maladies were caused by the exposure.
It’s worse still for those who hold statistical models—in which I include all artificial intelligence and machine learning algorithms—can discover cause. For what about all those people in either group who did not develop the disease?
Even if exposure causes disease sometimes, and the other (unknown-but-not-exposure) cause which we know exists only causes disease sometimes, we still do not know why they exposure or non-exposure causes disease only sometimes.
Why did these people develop the malady and these not? We don’t know. We can “link” various correlations as “cause blockers” or “mitigators”, but we’re right back where we started from. We don’t know—from the data alone—what is a cause and what is not, and what blocks these (at least) two causes sometimes but not in others.
Once again I claim over-certainty in medicine, and in epidemiology in particular, is rampant.
This is why I insist that cause is in the mind and not the data.
Why did these people develop the malady and these not? We don’t know. We can “link” various correlations as “cause blockers” or “mitigators”, but we’re right back where we started from. We don’t know—from the data alone—what is a cause and what is not, and what blocks these (at least) two causes sometimes but not in others.
You’re in the same boat when you have what might be a mechanism. Just because you think you have a mechanism you can’t prove it — even when you’re suspected mechanism is “linked”.
You really have described scenarios where you evaluated various two variables X and Y sets. You still need to introduce at least a third variable just as Pearl said.
Truth is, you will always use statistics to determine or confirm a cause through experimentation. And, if you don’t experiment, you’re guessing. Yes, sometimes you don’t need to run the numbers — a bullet in the head is harmful because, well, it usually is. Trying to determine whether cigarette smoking. on the other hand, causes lung cancer, e. g., is far trickier because all of the cases are at the noise level and many smokers survive to their 80’s while many who don’t smoke still suffer lung cancer.
Unfortunately, most of what epidemiology is finding nowadays is close to the noise level. It’s one of the reasons why we are bombarded with conflicting reports such as salt, sugar, wine, red meat, etc. is good AND bad for you. Well, that and p-values.
>Not everybody in group (a), the exposed group, will have developed the malady (or maladies; after which I use the singular to save typing), and some people in (b), the not exposed group, will have the malady.
Do you mean that the manipulation is a “Insufficient but Necessary part of a condition which is itself Unnecessary but Sufficient” for the outcome. (INUS condition)? I think most epidemiologists are familiar with Rothman’s component causation. You might be confusing epidemiologists with journalists and other casual causal commenters.
DAV,
Absolutely, yes, experimentation is a must. Direct, discrete, and at as a fine a level as possible, where one has removed what one believes all exterior causes. All cause in conditional as probability and logic are! Think of the fine physics experiments where we later learn deeper forces are at work.
My main point, besides over-certainty, is that cause is a something in the mind, not the data. Not that algorithms can’t help us discover it, but they can never provide proof. Where by “proof” I mean in its rigorous sense. Pearl’s “third variable” (etc.) can’t be be found by any machine, but by us. See that link about cause in the mind.
Bill,
Whether or not epidemiologists are familiar with the weaknesses and limitations of their approaches is a debatable point. Think of all the papers we have reviewed here over the years which claim cause—like PM2.5 and heart disease, like global warming and whatever malady you can think of.
I believe the problem is that, as you say, the limitations are taught and all acknowledge them, but they are soon jettisoned in the rush to publish. Recall what I always say: everybody believes in confirmation bias, but they always believe it happens to the other guy.
@DAV
“Unfortunately, most of what epidemiology is finding nowadays is close to the noise level. It’s one of the reasons why we are bombarded with conflicting reports such as salt, sugar, wine, red meat, etc. is good AND bad for you.”
The examples you provide are usually based on cohort studies, which are very, very low on the research methods totem pole. Asking people what they ate and when is not very reliable, is it? Based on a convenient cutoff, the groups are divided into ‘low/high’ salt intake or whatever. The latest craze are keto/paleo lunatics (high fat, low evidence).
But even in studies like that, when you acknowledge the limitations of how the data were collected and you read past the p values by inspecting odds ratios and other descriptive stats, you will notice that the evidence is rather small but it has certain direction. So if the magnitude of the effect is not to your liking, at least you have a general direction. I may not be able to isolate the effect of PM 2.5, smoking or over-eating among a myriad of variables, due to the lack of control or simply time, but those things are probably not good for you. Once Warren Buffet and other millionaires move into a cardboard box under the highway and start breathing polluted air, I might change my mind 😉
When it comes to multiple variables that affect the outcome (well, any outcome is affected by multiple variables, but some outcomes are primarily affected by only a few variables, while the others are harder to detect and ‘spread’ their causes over thousands of predictors) such as, PM 2.5, climate change, etc. it is not a yes/no answer one should be looking for but the extent of the evidence (the effect) and the direction. I may have a slight amount of evidence in the one direction, but it does not mean that because of my lack of a large effect, you can argue the direction is opposite.
See, you never hear of ‘diabetes deniers’, ‘stroke deniers’, ‘anaesthesia deniers’. Usually, the multiple variable problems that are hard to control methodologically are attacked.
BTW, the control/intervention group design that way described is the simplest one. There are various cross-over, double-blind, placebo-controlled, etc. designs that can address various issues.
In any case, p values shouldn’t be even looked at, but various metrics of the effect size should.
by inspecting odds ratios and other descriptive stats, you will notice that the evidence is rather small but it has certain direction
Uh. no. When the effect is small, the direction can be mere coincidence. Even when not, buying 20 Powerball tickets increases your chance of winning 20x but don’t quit your job waiting for that big payoff. Same with smoking. The CDC used to have a table on its site showing that the chances of NOT getting lung cancer from smokers were 98.96% and 98.98% for nonsmoking (not the specific numbers but in the right range — about 1 part per 10,000). The actual numbers were around 20x greater for getting lung cancer from smoking vs nonsmoking. Effectively, no difference at all just like with 20 Powerball tickets. The table has vanished from the CDC site. Why I wonder. Was it flat out wrong or did it not support the narrative? Not to mention that during the 70’s and 80’s (and maybe still) medical examiners began listing the cause of death as “smoking related” if they learned the deceased had ever smoked. When I went to school, skewing the numbers to fit the “known” answer was called fudging.
Too many epidemiology results use bad statistics (p-values, e. g., which prove nothing) and, at times, outright fraud to achieve the desired result. Just go to John Brignell’s Number Watch site for the long list of them. Few of them are cohort studies.
When it comes to diabetes and things like cholesterol have your noticed that the “acceptable” and “normal” limits keep narrowing each year? Why is that? What the hell is “pre-diabetes”? Is that someone who DOESN’T yet have diabetes — like being pre-approved for a loan (i. e., not yet approved)? Your body regulates cholesterol. To lower it requires medication. Who do you think pushes the lowering “normal” cholesterol limits? Cui bono.
DAV,
I’m not exactly sure how what you cite about smoking is different from what I wrote. If the chance of not getting lung cancer from smoking were 98%, that means the odds ratio they obtained in the study was very close to 1 (both outcomes are equally likely). The problem is that nobody bothers to understand what the various ratios (odds ratios are effect sizes) mean but go directly to p values for a yes/no answer.
Regarding ‘pre-diabetes’ and ‘pre-hypertension’; well, those are made-up categories where a perfectly continuous variable was categorized for the sake of simplicity. What I meant is that the outcomes in epidemiology are not up for discussion. People get strokes, heart attacks, diabetes complications, etc. There is no uncertainty connected with the outcome, only with the causes, which are multiple. The outcomes are loud, clear (and ugly) and you can count them precisely.
Also, when it comes to epidemiology (and statistics in general) you need to distance yourself from N=1 type of evidence. Nobody cares about that, as epidemiology deals with the outcomes at the population level. There is some evidence that inhaling smoke in any form is not good for you, which does not mean you can’t live long and healthy life while still smoking. It is about the population, not you or me. Overall, there is evidence (albeit small) in one direction and one direction only. You cannot demonstrate that smoking IS good and should be encouraged, just because the evidence for it being harmful was noisy and small.
I’m not exactly sure how what you cite about smoking is different from what I wrote.
Previously: by inspecting odds ratios and other descriptive stats, you will notice that the evidence is rather small but it has certain direction
I was showing that the direction is not a good measure either. The evidence that smoking causes lung cancer is non-extant or, at best, poor. It’s like saying you will win the lottery by buying 20 tickets instead of one. You’re a fool if you believe that. Buying 20 tickets does little to change your chances.
Regarding ‘pre-diabetes’ and ‘pre-hypertension’; well, those are made-up categories where a perfectly continuous variable was categorized for the sake of simplicity.
No. They are saying some normal values are worse than others. Downright silly. If the gauge is in the green then the value is OK. You either have diabetes or you don’t. What’s this “maybe” stuff?
There is some evidence that inhaling smoke in any form is not good for you, which does not mean you can’t live long and healthy life while still smoking.
So campfires are bad for you? What about fireplaces? You should shun being around them because you might inhale some smoke? And if you do what exactly is the harm?
Think about it, if someone can smoke two packs a day for 40+ years and not suffer from, say, lung cancer then how much less is the effect of being merely near a smoker?
when it comes to epidemiology (and statistics in general) you need to distance yourself from N=1 type of evidence. Nobody cares about that, as epidemiology deals with the outcomes at the population level.
??? Sounds like frequentist talk. Where exactly did I do this, anyway?
When it comes to N being small, I came across an epidemiology study where the groupings (cluster sizes) were N=4. So someone cared. Not that it was correct.
Epidemiology has entered the realm of diminishing returns with too many participants and not enough real things to find. Gone are the days of locating the water pump causing the typhoid epidemic. To survive, one must publish “findings” even when they don’t really exist or are at levels making the detection dicey at best.
You cannot demonstrate that smoking IS good and should be encouraged, just because the evidence for it being harmful was noisy and small.
Yet all things are toxic — even water. I’m not talking about drowning. Too much water is literally poisonous. Nothing is 100% safe. Should I avoid water because too much of it will kill me? What about eating red meat? How about candy?
Overall, there is evidence (albeit small) in one direction and one direction only.
So is sky diving or bungee jumping good for you? What about drinking alcohol? I used to smoke but lost the taste for it. So I stopped doing it. I still do drink alcohol on occasion. Are you saying all of those activities can’t be good for you? Why then do people do it? Are they insane?
What’s this one direction nonsense? Most people my age grew up around smokers and drinkers and didn’t think anything of it. It was the norm. I’m guessing you’re less than 50 and have been exposed to too much propaganda. You seem to be asserting that your tastes are paramount and everyone should accommodate you.
You cannot demonstrate that smoking IS good and should be encouraged, just because the evidence for it being harmful was noisy and small.
Yet all things are toxic — even water. I’m not talking about drowning. Too much water is literally poisonous. Nothing is 100% safe. Should I avoid water because too much of it will kill me? What about eating red meat? How about candy or french fries or a host of other things? Did you know that if your diet consists mostly of lean meat (say, rabbit) you will get protein poisoning? Does that make eating rabbit bad for you?
I’m saying smoking and all of those other things are not bad. If you don’t like them don’t engage in them. It’s that simple. If you do though, moderation is the key. Stop using bad statistics in an attempt to force everyone to be like you.
Pingback: Equality Is False — In Aptitude, Intelligence, Ability. What Are The Consequences? – William M. Briggs
What about this reasoning: exposure modifies (in some unknown manner) the constitution of the person exposed, in a way that facilitates the manifestation of the disease (whose cause remains unknown).