What would you name a paper which purports to discover that heat waves stress pregnant women and thus cause birth defects but which never once measures the actual exposure of pregnant women to heat?
If you answered “A Population-Based Caseâ€“Control Study of Extreme Summer Temperature and Birth Defects” then good for you, because just such a work was put into the peer-reviewed journal Environmental Health Perspectives (2012 October; 120(10): 1443â€“1449) by Zutphen and others.
Zuthphen and her co-authors examined birth records for a number of years in New York, plugging temperatures at a “crucial” time in women’s pregnancies and presence or absence of 84 different birth defects, along with a limited set of demographics into a statistical model and concluded:
We found positive and consistent associations with congenital cataracts of multiple ambient heat exposure indicators, including 5-degree increases in the mean daily [universal apparent temperature] (minimum, mean, and maximum), a heat wave episode, the number of heat waves, and the number of days above the 90th percentile of UAT.
The temperatures were measured at airports, and women were “assigned” the temperature of the nearest airport where they resided when they gave birth. Only 18 stations for the entire state of New York were used. This of course means we have little or no idea what temperatures any of the mothers actually experienced during their pregnancies (“we were unable to incorporate air conditioner use data…”). So we have no idea if excessive temperatures would have caused any birth defects for these women.
But statistical methods are immune to these kinds of subtleties; p-values can be produced no matter what relation a hypothesis has to actual data on hand. In this case, p-values were so produced, and for a couple of the 84 birth defects, the p-values were small enough to declare a “statistically significant” relationship.
Now this relationship is only with a few birth defects and temperatures at airports “near” where women lived when they gave birth, which is only loosely (at best) connected to the actual heat the women experienced during their pregnancies. But that’s too complicated to remember, so it’s easier to say, as the authors do say, “We found positive and consistent associations between multiple heat indicators during the relevant developmental window [of mothers] and congenital cataracts”.
I gave up early on this one and leave it as an exercise for long-time readers to discover the many other flaws of this paper. I’ll just mention the second biggest—the first is the use of the ecological fallacy, as detailed above—which I’ll put in the authors’ own words:
Under the null hypothesis, we would expect 4 of the 84 effect estimates displayed in Table 3 to be statistically significant at the p = 0.05 level. Thus, significant positive and negative associations with cataracts, renal agenesis, and anophthalmia may have been chance findings. Bonferroni adjustment to the p = 0.05 level of significance (0.05/84 = 0.0006) would yield approximate adjusted CIs for congenital cataracts that include the null value (95% CI: 0.93, 2.44).
In other words, after using the proper (frequentist) measure, the results are not statistically significant, meaning the conclusion stated above is false. As in, Nothing to see here. No effects were found. The results were insignificant.
But, darn it, they could have been significant, and that’s what’s really important. Climate change is that sneaky. So much so that the authors plead, immediately after admitting there was no significance after accounting for multiple testing, “However, the associations with congenital cataracts are biologically plausible”.
There you have it: they didn’t find an effect, but they could have, and that’s what’s really important.
Thanks to Dr K.A. Rodgers for alerting me to this paper.