CARB Misinterprets Statistics, Calls For Elimination Of Dust

The California Air Resources Board, or CARB, has issued a press release that shows how easy it is to misunderstand statistics. That most do—fail to comprehend what statistical results are and mean¹—is no surprise, but its particularly disheartening in CARB’s case.

Why? Because its misconstrual is followed by a call for increasing the size and scope of government. This exercise once again shows how willing bureaucracies are to latch on to any study that might, however tenuously, be in their favor. It is therefore a useful, if achingly dull, exercise to examine the press release and its history.

It begins with: “Fine particle pollution a threat to the cardiovascular health of Californians.” (Fine particle pollution is, among other things, dust.)

Three new studies released today by the California Air Resources Board reveal that exposure to airborne fine-particulate matter significantly elevates the risk for premature deaths from heart disease in older adults and elevates incidence of strokes among post-menopausal women. Heart disease is the number one killer in California and is responsible for approximately 35% of annual deaths.

Before we get to the meat, note that there will always be a “number one killer” so your antenna should always be up when somebody begins discussing what it happens to be this year. Secondly, notice the specificity of the claims. CARB isn’t stating that all citizens will develop heart disease or stroke after exposure to “airborne fine-particulate matter”, but that only some old people and post-menopausal women will (this latter group presumably will suffer both strokes and heart attacks). Specificity does not imply falsity of claims, but it is often the case the more specific the medical claim the more the suspicion that data snooping (searching for publishable p-values) has taken place.

That indeed is the case for one of the studies quoted by the press release: the statistical research of Michael Jerrett et alia. I reviewed in depth Jerrett’s study here and here. Briefly, here is what Jerrett did:

He gathered data on the nearness to possible sources of PM2.5 (the fine particulate matter) where Californians possibly lived at one point in their lives, then he noted whether the person died and of what by the study’s close. His original goal was to demonstrate that the greater PM2.5 exposure lead to higher death rates. This goal failed, in the sense that his statistical researches did not generate enough evidence to say so.

But the data he collected allowed him to test multiple causes of death, so he searched until he found one cause among the many. This cause produced publishable p-values. He never adjusted for the multiple testing, which always gives p-values smaller than they should be. But never mind that. And forget that the sources of pollution and cardiac death rates differed by an order of magnitude between urban and rural populations and that the model Jerrett used did not adequately account for this, therefore improperly inflating his confidence.

Ignore these and all the other mistakes and focus on the conclusion, which stated that “exposure to fine particulate matter significantly elevated the risks for premature death from heart disease.” But this is not what the kind of statistical models Jerrett used can say. He never measured the actual PM2.5 exposure of even one person. There is no causality, only suspicious correlation in his models. And these models say nothing directly about even the correlation: they instead make statements about functions of certain parameters in the model.

The press release should have read: “Small unadjusted p-values were found in one of several models that say a certain parameter associated with once possibly living near a source of dust might be correlated with a fractional increase in the possibility of heart disease.” A mouthful, but true and justifiable, unlike the actual headline which possess the opposite of these traits.

Incidentally, I and several others gave extensive critiques to CARB, which accepted and discussed them at an open meeting. My comments were considered, but ultimately dismissed by the CARB panel which argued that the errors I identified were of the sort common in studies like Jerrett’s, and that since these other error-ridden studies had been accepted, so should Jerrett’s.

This fallacy is indefeasible. It is also rife, the essence of bureaucratic scientism. It is the grown-up’s version of the teenager’s everybody-else-is-doing-it argument. The if-everybody-else-jumped-off–cliff retort is nevermore heard.

The press release kicker: CARB Chairman Mary D. Nichols reasoned that “These new studies underscore the need to eliminate the threat [of dust] from California’s air.” The only threat that has been demonstrated is Nichols’s anxiousness and determination to burden us with more rules and regulations.

—————————————————————————-

¹The fault lies with we statisticians, and begins in the classroom.