A Case Of Failed Peer Review: Dust And Death

The distance between what civilians think peer review is and what it actually is suffers from the same failing as that evinced by Han Solo—rare pop culture reference!—when he boasted to Obi Wan Kenobi that the Millennium Falcon could do “the Kessel run in less than twelve parsecs.” Let him that readeth understand.

Peer review—an institution a bare century old, and arising solely to control the page count of proprietary journals—is the weakest filter of truth that scientists have. Yet civilians frequently believe that any work that has passed peer review has received a sort of scientific imprimatur. Working scientists rarely make this mistake in thinking.

Here is an example of how the peer review process works—or rather, does not work.

The California Air Resources Board (CARB) met on 28 October to discuss the Jerrett report “Spatiotemporal Analysis of Air Pollution and Mortality in California Based on the American Cancer Society Cohort: Final Report (as revised)” by Michael Jerrett, Richard T. Burnett, and a host of others.

This is a study which claims to have found a statistical—not actual—relationship between dust (PM2.5) and premature death for (at least part-time) California residents. I reviewed this paper and found several significant flaws in the use and interpretation of statistical methods. Here is the most significant: “I find further that the summary in the abstract—and therefore the only part of the report liable to be read by most—to be the result of either poor work or deliberate bias toward a predefined conclusion.”

The authors prepared and intensely investigated a series of complex statistical models. There were nine models in total, each having particular strengths and weaknesses. Each had several subjective “knobs” and “dials” to twist. Only one model of the nine (p. 108) showed a “statistically significant” relationship between mortality and PM2.5, and that only barely; and in that model, only one sub-model showed “significance.” The other eight models showed no relationship. Some models even hinted that PM2.5 reduced the probability of early mortality. With such a large number of tests and “tweaks”, the authors were practically guaranteed to find at least one “significant” result, even in the absence of any effect. Nowhere did the authors control for the multiplicity of testing, even though such controls are routine in statistical analyses of these sort.

You may even listen to the CARB meeting (mp3: 80 minutes).

There were seven separate critiques presented, all by peers with significant and lengthy expertise in relevant areas. Comments were provided by: Jim Enstrom, Matthew Malkan, John Dunn, Frederick
Lipfert
, Gordon Fulks, Skip Brown (Delta), and yours truly (updated). A quote from each:

  • Enstrom: “The results in the Jerrett Report do not support the authors’ claim.”
  • Malkan: “[The] Abstract, Key Results, Key Findings, and Conclusion sections which do not accurately reflect, and are even contradicted by, the actual data analysis presented in this report.”
  • Dunn: “[W]e have a modeling paper that looks a lot like the nonsense put out on global warming modeling, and it has the taint of data torturing in its presentation.”
  • Lipfert: “I find that the consistent and overwhelming defect in this report is its arbitrary selectivity:…Selecting heart disease as the most important cause of death, while ignoring the apparently significant beneficial relationships with cancer.”
  • Fulks: “With the apparent approval of the agency staff, the authors have refused to correct or even address mistakes.”
  • Brown: “This ‘new’ report, whose entire purpose is to justify previously passed regulation, does not address the many scientific comments made rebutting the conclusions reached in the original report.”
  • Briggs: “I find further that the summary in the abstract—and therefore the only part of the report liable to be read by most—to be the result of either poor work or deliberate bias toward a predefined conclusion.”

CARB had earlier implemented regulations based on the assumption that particulates kill. The story of how they came by that assumption is odd, but is not relevant here. The Jerrett report was meant to bolster the research that led to the regulations that were already in force. Therefore CARB was to decide only whether to accept or reject the Jerrett report. Despite the numerous flaws and objections given by Jerrett’s peers, after a few minutes discussion CARB voted to accept the report. In one sense, this was fine, because without this acceptance Jerrett could not claim that he fulfilled his contractual obligations.

But in the sense of approving the findings themselves, this peer review process clearly failed. This is true even if the large numbers of criticisms were wrong or inconclusive. This is because rebutting serious criticism takes time, thought, and effort. CARB did not attempt to rebut any of the criticism beyond saying because what Jerrett claimed was also claimed by other authors, therefore Jerrett’s findings should be accepted. Another commentator said that because science is imperfect, we may as well accept Jerrett’s findings.

Then the criticisms were not wrong, especially the “cherry-picking” critique cited above. The statistical mistake (choosing only the significant model which showed “significance” and ignoring the ones that did not, and for not correcting for multiple tests) made by Jerrett is enormous, and if addressed would have caused the claim of “statistical significance” to disappear. It is thus more likely that what Jerrett claims is false.

This is a (not at all unusual) failure of peer review.

Update My critique was commented on starting at 47 minutes in.

20 Comments

  1. DAV

    There is most certainly a relationship to dust after death. Even the Bible mentions it as in: “dust thou art, and unto dust shalt thou return” from one of the many versions.

    So CARB has taken the EPA approach and proceeds along its chosen path simply because they can. Flexing muscle is a good thing for any bureaucrat. If the science agrees then so much the better. Why are you surprised?

  2. Speed


    Another commentator said that because science is imperfect, we may as well accept Jerrett’s findings.

    And because they are spending other peoples’ money.

  3. GoneWithTheWind

    The real problem with these guesstimations is that they use the same deaths to justify a lot of oppressive and anti-constitutional laws and programs. I’m sure that some of those people who died from dust also died from global warming, wood stoves, lawn mowers, flouride in the water, lack of Obamacare, etc. What we need to fix this obvious statistical fraud is more deaths so we can spread them around better and maybe even get more oppressive laws passed. I’m making a bad joke of course but I am concerned that many/most of the repressive/oppressive laws and programs will result in more deaths.

  4. I’m confused. The Jerrett report appears to have been produced under contract to CARB. I don’t think the average layperson (me, for example) thinks of peer review in the scientific sense as the result of a bureaucratic committee commissioning a “scientific” study and then, ostensibly based on their understanding as modified by input from commenters, accepting or rejecting the study. This process is obviously flawed, as you’ve clearly demonstrated but, as I said, I don’t think many would consider this to be an example of peer review gone wrong.

    I think of peer review as the process by which articles, studies, experiments, etc. submitted for publication in scholarly journals get vetted by experts in the field with which the article is involved. While this process is, as best I can determine, criticized by most of its participants as being much less than perfect it is also thought to be fixable and the best method currently available to separate wheat from chaff.

    All that said, when I think of peer review and its flaws, the process you’ve described here (as admittedly awful as it is) is not what I’m thinking of. Feel free to enlighten me.

  5. DAV

    GTW,

    I think the real reason behind CARB is money (or maybe i’s just a fortunate byproduct). The city I live in passed some rather strange laws recently under the banner of Safety but the real reason was to implement money traps.

  6. JH

    I am puzzled… what peer review process has the report undergone? Has it been sent to a journal for publication? Sounds like it’s just a report to CARB and contains a bunch of co-authors with unknown contributions to the report.

    The peer review process never stops at the acceptance of an article by a journal. Anyone interested in a certain paper is welcome to review it and make corrections. So, if anything, it works because you’ve decided to review it for whatever reason. In a way, your review is part of the peer review process.

  7. Grzegorz Staniak

    Mr. Briggs,

    You’re talking about a report comissioned by a state office and stamped with “Review Draft” all over. This is a very bad example of peer review in any case, and the statements that you pass as proof of bad review are exactly the process of review of a comissioned report. That’s at least a bit misleading.

  8. Briggs

    Rob Ryan,

    Good questions, which highlight more areas where civilians and scientists see the process differently.

    Peer review is not the exclusive purview of proprietary or member-controlled journals. The government often uses it; for example, to award grants or upon which to base legislation. If you listen to the panel discuss this case you will hear other peers in support of or in the disagreement of the Jerrett report. All offer evidence one way or the other.

    My only claim is that there was not sufficient time to consider the evidence and that the critiques offered were of such importance to warrant more review.

  9. Ray

    if dust killed then I should be long dead. I grew up in New Mexico and we had bad dust storms. The blowing sand would pit the windows and paint on cars. We used to joke that Arizona blew thru NM on the way to Texas in the spring and in the fall Texas blew thru NM on the way to Arizona.

    How do you determine if a death is premature?

  10. TMI

    Why worry about the models, when it is the results that matter.

    Apparently CARB got the results it paid for.
    .

  11. Speed

    From the California Dump Truck Owners Association, March 14, 2011 …

    We are continuing to aggressively compel CARB to respond to CDTOA’s Public Records Act request for unredacted and updated information relating to the $750,000 CARB/SCAQMD-funded study by UC Berkeley Professor Michael Jerrett called the “Spatiotemporal Analysis of Air Pollution and Mortality in California Based on the American Cancer Society Cohort.” This publicly funded study was supposed to be completed at the end of March 2010, a year ago, and we are still finding it difficult to get the latest progress report and explanation as to why the report is late—a year late!

    If there is no proof, as we suspect, then diesel should be removed from the list of TAC’s [Toxic Air Contaminants].

    Then the key members of the SRP should be subpoenaed and hopefully indicted on fraud, a pattern of academic and health science racketeering and exclusion to benefit themselves and their publicly funded schools. We believe much of this could be challenged under the RICO act, maybe even a Bivins suit.

    They seem upset.

  12. pouncer

    Should I tell my own experiences with the CARB I would be at risk of losing my job. Let me hint that these experts in California make regulations without regard to how businesses of the 21st century actually operate. Note: We do NOT keep invoices in file cabinets near the receiving dock any more…

  13. David L. Hagen

    Does this discussion of the Kessel run help in the understanding?

    By moving closer to the black holes, Solo managed to cut the distance down to about 11.5 parsecs

  14. Briggs

    David,

    Too much time on your hands!

  15. commieBob

    This is totally off topic (related to statistics though) but I saw this over on Coyote blog:

    If you choose an answer to this question at random, what is the chance you will be correct?

    A) 25%
    B) 50%
    C) 60%
    D)25%

    The origin of the question was at: http://flowingdata.com/2011/10/28/best-statistics-question-ever/
    When I checked it, there were 369 comments, one of which invoked string theory. Any thoughts?

  16. DAV

    commieBob,

    Today, on the most Hallowed of Weenies, you have come up with the scariest post.

  17. Speed

    From the Freakonomics Blog …

    We present the gender ratio of live births as an under-exploited metric of fetal health and apply it to examine the effects of air quality on fetal health. Males are more vulnerable to side effects of maternal stress in utero, and thus are more likely to suffer fetal death due to pollution exposure. We demonstrate this metric in the context of the Clean Air Act Amendments of 1970 (CAAA) which provide a source of exogenous variation in county-level ambient total suspended particulate matter (TSPs).

    We find a statistically and economically significant association between ambient TSP levels and the fraction of live births that are male …
    NBER Working Paper (abstract)

  18. Briggs

    commieBob,

    I love it. Perhaps I’ll do an extra post today in, as DAV says, honor of Halloween.

Leave a Reply

Your email address will not be published. Required fields are marked *