Video: The Crisis Of Evidence, Or, Why Probability & Statistics Cannot Discover Cause

Thanks to Jeremy Snavely at DDP for putting up the talks. Here’s a direct link. Here’s the paper.

Q & A starts around 52:30 minutes. Since some readers were curious, George Gilder questions me about falsifiability around 56 minutes.

Yes, I was aware how close to the edge of the stage I was. It was a small platform. I’m used to running up and down the aisles waving my arms around like an ape, so I felt rather constrained here.

And as long as we’re at it, I’m no fan of “slides.” I try to have few to no pictures or other aides other than those that are essential. It’s a mystery why any would want to have a picture that says boldly, “These are the words I’m saying” while I’m saying the words that they’re saying.

I have learned that businesses now call groups of slides “decks.” They speak of “working on decks”, and this is true. Vast manpower is given over to perfecting PowerPoint pictures that are shown only once or rarely. The scourge of WYSIWYG.

20 Comments

  1. Larry Geiger

    Those “slide decks” become entombed. In government speak, they are “RECORDS”. The “RECORDS” custodian knows all of the arcane rules that apply to all types of “RECORDS”. They must be kept a certain way and a certain number of years. If they reach a certain level of importance they must be printed out and placed into a file cabinet. Anyone here who has worked with a “RECORDS” cusodian will understand the pain.

  2. Nate

    I’m in consulting, and they love decks. They sum up hard things into easy to digest bullets for the ignorant managers. Nobody wants to think, they just want to look at a pretty picture and be reassured that X will solve all their problems. Plus since its on a slide you don’t have to write difficult sentences that explain the buzzwords.

  3. Phenomenal presentation. Love the Bayesian reference. There are a small group of people who seem to think the same way. We need to stick together.

  4. MattS

    “Nobody wants to think, they just want to look at a pretty picture and be reassured that X will solve all their problems.”

    And the fact that X wont solve any of their problems is irrelevant.

  5. Leo

    Thank you for making the talk available.

  6. JohnK

    In the abstract, should be:
    Classical statistical procedures, in both their frequentist and Bayesian implementations, falsely imply they can speak about cause.
    (no “are”)
    Also:
    This over-certainty is made much worse when parametetric and not predictive methods are used.
    should be “parametric”

  7. Gary

    Slides are useful for showing pictures. Keep the words to a minimum. I think too many presenters use them as note cards because they don’t have the skill to talk with their audience and need a crutch.

  8. Francsois

    I enjoyed the talk Briggs, thanks. Every medical paper I have ever read always say “associated with” or a similar term. None of them say because they found p value less than 0.05 that they are certain that x causes y. They only say that it is possible, perhaps even likely in light of other research and plain reasoning. In medicine decisions have to be made because lives are at stake. So the evidence using p-values is not perfect, but it is likely to be better than accepting the opinion of experts (I think). Example: imagine you observe a group of people. Some of them get shot in the face with an assault rifle, and some get soap bubbles blown onto their faces. You are likely to notice a difference in the number of deaths between the bubble and assault rifle group, with whee p values. This does not PROVE that shooting someone in the face with an assault rifle is responsible for the deaths, but, the p values taken in context of physiological reasoning and common sense does strongly suggest that shooting someone in the face causes death!

  9. Nate

    Francois, isn’t the null hypothesis in your experiment “shooting people in the face will cause no harm”? Aren’t you supposed to actually *believe* the null?

  10. DAV

    Francsois,

    The p-value, at best, gives you the likelihood that the difference between the groups in the data you’ve collected is not a figment of your imagination. It tells you nothing else.

  11. Francsois

    Hi Nate, why would have to believe the null? Dav, “likelihood that the difference between the groups in the data you’ve collected is not a figment of your imagination”. That sounds pretty good to me! I can go with that.

  12. davideisenstadt

    but noting that a difference exists between two descriptive statistics tells you nothing about wther there is some causal nexus between the two.
    You dont need to do any kind of statistical analysis to determine that getting shot in the face causes harm
    We already know the physical mechanism responsible for your wee p value…
    why not tackle an issue like gender and incarceration rates…
    Can your analysis tell us anything about why they disparity exists?
    really.
    What about hem lines and GDP growth…care to posit a causal link between the two that makes some sense to us?

  13. Francsois

    You might find a association between hemlines and GDP growth…. But because there is no sensible way to explain, and since it is rather silly, one can dismiss it for now. Just because you cannot prove causation with associations, does not mean it is clever to ignore associations either. Sometimes we find associations that do not have a preceding basic scientific explanation. The association prompts us to look at possible reasons for the association, i.e. we investigate causal mechanisms with basic science etc. Cigarette smoking is a good example. Associations were found between smoking and death. Nobody really understood the pathophysiological reasons for the association, but it seemed plausible, and was investigated further. Do you believe that smoking causes cancer David? I agree that associations can be abused, as in Briggs PM 2.5 example. But it would be silly to ignore associations.

  14. Francsois

    ” but noting that a difference exists between two descriptive statistics tells you nothing about whether there is some causal nexus between the two.” I know this, and said as much: “None of them say because they found p value less than 0.05 that they are certain that x causes y. They only say that it is possible, perhaps even likely in light of other research and plain reasoning.”

  15. DAV

    They only say that it is possible, perhaps even likely in light of other research and plain reasoning.”

    But it’s that likely, often without any basis other than an association, that’s the problem. In a surprising departure from political correctness**, Scientific American, published It’s Time to End the War on Salt

    All too often, a paper jumps to a conclusion either directly or by implication in press releases. Or even worse, there is often failure to correct public misinterpretations or the paper. Follow up studies rarely test if said “cause” is genuine but instead do “let’s see if the association holds” testing. So, the only certainty is that the association holds. There is a proposed mechanism for salt’s, er, badness but to date, no one has really tested if it’s true.

    An interesting quote from the SciAm article:

    For instance, a study published in February 2010 in the New England Journal of Medicine estimated that cutting salt intake by about 35 percent would save at least 44,000 American lives per year. But such estimates are not evidence, either; they are conjecture.

    Alderman and his colleague Hillel Cohen propose that the government sponsor a large, controlled clinical trial to see what happens to people who follow low-salt diets over time.

    One can only wonder how those lives-saved estimates were derived if “cause” was only possible.

    Salt isn’t the only example. There are plenty more. It’s stuff like this that Briggs is railing against. There is far too much leaping from p-value hypothesis testing to causes of X.

    ** then blew it in the See Also section. Aw well.

  16. francsois

    DAV, I agree with you “There is far too much leaping from p-value hypothesis testing to causes of X”. I am saying that we should not dump association studies, just do things properly. The salt association example you used: they should have followed up the epidemiological investigations with a trial, or with basic science investigations to find out how salt works to kill people. Not always easy to follow up with a trial, take smoking as an example. Thanks for the reply.

  17. Shack Toms

    The interesting thing to me about bureaucrats deriving power through low p values is this. Given that p reflects the proportion of studies that would yield a no less extreme statistic, even if the null hypothesis were true, then experiments can be designed so that approximately 1 in 20 will have sufficiently low p, even if cause is not involved at all. So all they have to do to get more power is to fund more studies. They don’t even have to cook the results.

    This could easily become a “show me the man, I’ll show you the crime” situation.

Leave a Reply

Your email address will not be published. Required fields are marked *