Epidemiology is nothing if not a productive field. All that is needed for success is a database (larger the better), a disease (any will do), and some minor facility with statistical software.
Our latest example is the Environmental Health Perspectives1 paper “Residential Proximity to Freeways and Autism in the CHARGE Study” by Volk et al.
The authors found a group of mothers who lived in California. They measured the distance these mothers lived to “freeways and major roadways” for the majority of their pregnancies. They also took note whether their children developed autism. They posited that living closer to freeways increased the risk of autism. They also measured mothers’ education, age, and smoking status, the kids’ race and whether the kids were preemies.
They purposely identified 304 kids with autism and 259 without from a database “frequency matched by sex, age, and broad geographic area.” Ideally, since this data was hand-picked, they should have had equal numbers in each group, and equal frequencies of boys in each group. But the autism group had 87% boys, while the normal group had 81%. In other words, by design (purposeful or accidental), they put more boys in the autism group than they put in the control group. They gave this difference a “Chi-square p-value” of 0.10. What does that number mean? Well, nothing (see the footnote2).
As to the models:
Specifically, we included child’s sex and ethnicity, maximum education level of the parents, maternal age, gestational age at birth, and maternal smoking during pregnancy.
This was a logistic regression model, which here assumes the log-odds of developing autism is a linear function of the attributes just mentioned plus the distance (in meters) from freeways or major roadways. No plots of the freeway or roadway data are shown which indicate if this is a good assumptions.
Instead, the authors do a strange thing: they do not model the actual distance but chop up the distance into arbitrary bins. The first is living less than 309M from a freeway (about three football field’s distance). The next is living from 309M to 647M, then living from 647M to 1419M, and finally living greater than 1419M. They did the same thing for major roadways: less than 42M, 42-96M, 96-209M, and greater than 209M.
Two separate models were run: one for freeways, the other for major roadways. Only the less-than-309M group with respect to the greater-than-1419M group reached (classical) statistical significance. None of the other groups did. Nor did any bin in the roadways model. The (exponentiated) parameter associated with the freeway 309M-model was 1.86. This is incorrectly said to the the odds ratio for those to live withing 309M compared to those living farthest. It isn’t: it’s the parameter. To get the real odds, we’d have to “integrate out” the parameter, which would make the real odds ratio, assuming all else true and good, to be less than 1.86.
Remember when we talked about how changing the start date in time series analysis can lead to opposite conclusions? It’s the same here: why 309M and not, say, 308M or 310M? And the same for the other buckets. Different cuts will give different conclusions. Why not just leave distance in as a linear function? I mean, why chop it up at all?
Be generous and assume that these cuts are “real” and the “best”. Can we think of any other reason which might account for the results? Living within a football field or two of freeway in Los Angeles is a good indicator of what? Great medical care? Wealth? Health insurance? You’ll notice the authors left out any measure of economic importance.
In other words for this study, the effect is small, it is only for one small suspiciously chosen subset of the population (10% by the authors’ reckoning), and the posited cause is most likely an artifact caused by mismeasure and conflation of unmeasured socioeconomic variables. In short, the article gives no more than a vague suspicion that freeways are autism inducers: it even says they usually are not. It also says roadways are not autism inducers.
What makes this study interesting, then, is how it was reported in the press. The Wall Street Journal, not usually given to flights on fancy, reporting on this paper (and others) led with the headline:
The Hidden Toll of Traffic Jams
Scientists Increasingly Link Vehicle Exhaust With Brain-Cell Damage, Higher Rates of Autism
Lots of reasons given how exhaust might influence this or that biologic process, words like “Scientist believe”, a quote from the study author (“The evidence is growing that air pollution can affect the brain”), a quote or two from non-authors (“There is real cause for concern”).
CBS news, not content with the actual numbers, juiced them a little: “A new study shows that children in families who live near freeways are twice as likely to have autism as kids who live off the beaten path.”
No news source I could discover provided any analysis of how weak—and even nonexistent—the effects of this study were. Lesson for reporters: don’t trust scientists. We are no different than anybody else.
1Volume 119, number 6, June 2011, pp. 873–877. Thanks to Willie Soon for suggesting this topic.
2Ordinarily, in frequentist statistics, a chi-square test is used to test for “differences in proportions” in groups. Here, there were two groups with proportions 87% and 81%. Are these different? This is not a trick question, but it also one which is not to be answered within classical theory. That is, the chi-square is not an answer to this obvious question. The test is not a test of difference in actual proportions, but something else. Okay?
Instead, the statistician asks, “Assuming the ‘true’ proportion of boys with autism and boys without autism is identical: if we sampled from these two groups indefinitely, what is the chance of seeing a certain mathematical function (the chi-square) of these two sampled proportions being larger than the chi-square we see for the actual data?” This is 10%.
And so? Well, again, well nothing. The statistic has no bearing or meaning to this data. The database was built by hand with the intent of matching by frequency the sex of kids with autism. It failed in this; slightly, but it still failed. The authors could have, just as they picked the other data by hand, tossed out a few of the boys in the autism group or added a few more in the control group. There was nothing “random” in these selections, not even in the classical sense.
Now this dull subject is important because, as all prior evidence indicates, boys are vastly more likely to develop autism than girls. Why this is so, while interesting, is not relevant to this study. Why is relevant is that this discrepancy, the actual difference in proportions in the sample, might account for the “significance” in the results even though the authors included sex in their models.