Our friend Christos Argyropoulos (â€@ChristosArgyrop) to a popular medical site in which Stephen Reznick asks “Keep statistics simple for primary care doctors.”
He can’t read the journals, because why? Because “Medical school was a four year program. The statistics course was a brief three week interlude in the midst of a tsunami of new educational material presented in a new language…While internship and residency included a regular journal club, there was little attention paid to analyzing a paper critically from a statistical mathematical viewpoint.”
Reznick has been practicing for some time and admits his “statistical analysis skills have grown rusty…When the Medical Knowledge Self Assessment syllabus arrives every other year, the statistics booklet is probably one of the last we look at because not only does it involve re-learning material but you must first re–learn a vocabulary you do not use day to day or week to week.”
What he’d like is for journals to “Let authors and reviewers say what they mean at an understandable level.”
Now I’ve taught and explained statistics to residents, docs fresh out of med school, for a long time. And few to none of them remember the statistics they were taught either. Why should they? Trying to squeeze a chi-square test among all those muscles and blood vessels they must memorize isn’t easy, and not so rewarding either.
Medical students learn why the ankle bone is connected to the ulniuous, or whatever the hell it is, and what happens when this or that artery is choked off. Useful stuff—and all to do with causality. They never learn why the chi-square does what it does. It is presented as mystery, a formula or incantation to invoke when the data take such-and-such a form. Worse, the chi-square and all other tests have nothing to do with causality.
A physician reading a journal article about some new procedure asks himself questions like, “What is the chance this would work for patients like mine?”, or “If I give my patients this drug, what are the chances he gets better?”, or “How does the cure for this disease work?” All good, practical, commonsense queries.
But classical statistics isn’t designed to answer commonsense questions. In place of clarity, we have the “null” and “alternate” hypotheses, which in the end are nothing but measures of model fit (to the data at hand and none other). Wee p-values are strewn around papers like fairy dust. What causes what cannot be discovered, but readers are invited to believe what the author believes caused the data.
I’ve beat this drum a hundred times, but what statistical models should do is to predict what will happen, given or conditioned on the data which came before and the premises which led to the particular model used. Then, since we have a prediction, we wait for confirmatory, never-observed-before data. If the model was good, we will have skillful predictions. If not, we start over.
“But, Briggs, that way sounds like it will take longer.”
True, it will. Think of it like the engineering approach to statistics. We don’t rely on theory and subjectively chosen models to build bridges or aircraft, right? We project and test. Why should we trust our health to models which have never been been put through the fire?
One benefit would be a shoring up of the uncertainty of side effects, especially the long-term side effects, of new drugs. Have you seen the list of what can go wrong when you eat one of these modern marvels? Is it only us civilians who cringe when hearing “suicide” is a definite risk of an anti-depressant? Dude. Ask your doctor if the risk of killing yourself is right for you.
What the patient wants to know is something like, “If I eat this pill, what are the chances I’ll stroke out?” The answer “Don’t worry” is insufficient. Or should be. How many medicines are released only to be recalled because a particular side effect turned out more harmful than anticipated?
“Wouldn’t your scheme be difficult to implement?”
It’s a little known but open secret that every statistical model in use logically implies a prediction of new data. All we have to do is use the models we have in that way. This would allow us to spend less time talking about model fit and more about the consequences of particular things.
“What are the chances people will switch to this method?”
Slim.
Medical education is not the same everywhere.
In many institutions, there is a 1 and a 1/2 year block dedicated to 3-4 core subjects, one of which is public health. It goes by different names in different programs. Medical students going through these programs gain knowledge in epidemiology, statistics and population medicine topics even if they did not like the subject.
If you do post-graduate training, medical education is not the same everywhere. In Commonwealth countries, it is customary for there to be submission of a thesis. While this is not as intensive as a PhD thesis and many clinical disciplines don’t like it and view it as a burden, and, the quality of the product is extremely variable, there is exposure to doing science and analytic thinking. Most of these are observational and cross-sectional studies but they are there nonetheless. American medical schools do not incorporate this in their training.
Briggs’ comment on side effect of drugs was interesting. I had a neighbor who read the possible side effects and was horrified. He needed the medication, however, so I had to explain why those effects are listed. I once called a drug company to find out how a side effect (I think it was menstrual cramps) could possibly occur from the use of the drug. They explained that there was a set threshold of 2%, if I remember correctly, for reporting side effects. If the study had 2% of participants with cramps, that became a side effect. Same for flu or any other weird thing that occured 2% of the time. The person I spoke with understood how foolish this sounded, but it was the rules. If you also consider how many people, on a percentage basis, react negatively to foods and other products, where these reactions are not monitored, you realize that everything in life has risks (that’s what keeps personal injury lawyers in business) and the drug is possibly no more risky than the apple you’re about to eat or the steak you’re grilling.
I advocate educational reform after which:
Sixth grade: We teach all students the classical logic.
Twelfth grade: We teach all students measure theory and probability theory.
Fourth year of college: We teach all students information theory, theorizing and the scientific method.
Unlike a dissected cadaver, the chi-square test is intangible. So is it any wonder that its meaning doesn’t stick in the memory as well as the memory of arteries and muscles? It’s for this reason that simple arithmetic is taught to young children using “manipulatives” — small objects — that they can count and parcel out into group. While graphs are somewhat useful for illustrating statistical models, they’re still static attempts to convey change over a range of values. The problem of understanding that most of the population has would be helped greatly if the statistical models were represented dynamically and more tangibly.
Some random thoughts. Yes, chance made me do it.
A good practitioner of martial arts knows the ankle bone should be connected to the chin bone.
Χ² resembles the blades in a disposal. After using it to chop up your data you can pick out the best parts.
—
Have you seen the list of what can go wrong when you eat one of these modern marvels?
The cynic within me thinks the pharmaceutical industry relishes these opportunities for more drug sales.
DAV,
Big Pharma lobbied Congress hard about 20 years ago to let them advertise on TV and about a third of the cost to consumers pays for it. My guess is that Congress required the list of side effects to make everything look legitimate.
Briggs, kudos to your doctor friend for inquiring about statistics. Perhaps “keeping it simple” is not in the best interest of our patients though. Understanding statistics is just as important as say understanding pharmacology or diagnostic skills. Who wants to go to a doctor that strives to just have a “Simple” understanding of pharmacology, or diagnostic procedures? I’m not saying physicians need to have a Ph.D. level statistics education (although that would help). After all we need to leave room for great statisticians like you.
In medical practice there are several statistical areas, that come to mind, that any competent physician must know.
1) Test outcomes, knowing and communicating Positive and negative predictive value of any test or diagnostic consideration. This is perhaps the most common consideration in allopathic medicine. We use this knowledge daily in order to prevent false hopes or anxiety deciding which tests to employ, and who to communicate the findings to our patients.
2) Diagnostic concerns, communicating the chances of having/ not having disease “A” versus “B” given presence or absence of signs and symptoms “X, Y, Z.”
3) Continuing Medical Education, specifically understanding medical research to see if researchers really do “prove” or find what they say they do. Sadly this skill is sorely lacking in many practitioners, and is surprisingly deficient in some academicians. This last component is crucial to every physician’s practice in deciding changing strategy, treatments, potential outcomes, risks, drug therapies, all in a context around evolving knowledge base in medicine. Many academic articles, as you have so often pointed out fail miserably, to actually address the stated goals of therapy. Many docs just go with the latest therapy, or the one whose pharmacy reps give the best lunches or parties.
I am sure there are many more areas, but I am not feeling too well just now, and these just came to mind.
Doctors as a group are perhaps more intelligent than the population as a whole. However there are many doctors who just get by. We, like many become mediocre and fail to realize or ask about what we don’t . that is we tend to suffer from the Dunning-Kruger Effect.
Unless we ask, observe others in our field, including statisticians, and constantly strive to find out what we don’t know, we will not improve or skills. But more importantly our patients will not receive the best medical care.
As a student in a medical field, I have to read lots of research articles. Can not tell you how often I see something like…”a tendency towards ‘X’ but not statistically significant. ” My old stats prof would have torn out tufts of hair seeing that.
I go by a rule of thumb reading medical research: If the effect is so small that it requires meta-analysis of many conflicting studies to find, it probably isn’t relevant in the real world.
By the way, my first stats class was in the College of Agriculture. They were teaching us how to design field trials for crops. It was fun, and we got to grow fields of garbanzo beans for our trials.
>It’s a little known but open secret that every statistical model in use logically implies a prediction of new data. All we have to do is use the models we have in that way. This would allow us to spend less time talking about model fit and more about the consequences of particular things.
>What are the chances people will switch to this method?
It strikes me the answer to this used to be “possible” to “very good”. Because some to many of the ‘people’ switching to this method would have been the actual consumers of ‘health care’ in the U.S. ; which is to say, some to many of the actual payers of the bills of ‘health care’ — not us, but rather, private health insurance companies in at least semi-competitive markets. And they would have been very interested in more factual answers, straight up, and unashamed to require them.
But now, those companies, their uncomplaining mouths stuffed with gold for a few more years on the way to their oblivion in single-payer, are just one more needy part of USG.
So Matt’s ‘slim’ might be over-optimistic now.
One aside. It’s vaguely charming to read physicians’ comments that yes, they do have, or at least, ought to have, the time (alone), plus the statistical expertise, to critically analyze studies that, as Matt has forever pointed out, are — the best of them — chronically over-certain down to the bone, to begin with, and let’s not even get into the should-be-notorious failures to replicate ‘gold standard’ research, the (statistically) blind leading/teaching/’peer’ reviewing the blind, etc.
No, dear doctors of medicine, you should not need to calculate the square root of the universe in order to practice medicine well. Bad enough that a physician must learn, on his own, how to run a practice, or fit in to a massive institution.
In your ‘spare time’, take a nap, enjoy your family. You don’t have the expertise to solve this problem, and developing that expertise is very expensive, both in terms of your time and energy, and in the opportunity medical cost to us, your patients.
You shouldn’t have to be afflicted with this, and it would be a sad waste of your time to try to fix it yourself. Like your patients, you just need solid information, presented accurately.
Like Matt, I wish I could see someone in the current US system that could plausibly provide the systemic monetary incentives to obtain what you and we both need.
“What are the chances people will switch to this method?†Slim –
But Ben Goldacre is making a valiant try, I think, (in medicine, though not in CAGW) with his new book –
http://www.badscience.net/2014/10/my-new-book-is-out-today-here-is-the-introduction-hooray/
Laudable comments:
A friend of mine, a medical doctor, was on duty one night when a patient ripped a Foley Catheter out of his body and came walking down the hallway.
“What should I do?” He asked not having had to deal with this issue before.
“Call Urology,” said a nurse.
He called and ‘urology’ said, “Push it back in.”
He did and things improved.
Back to the issue at hand.
It would help if students took logic and understood probability and uncertainty and never made the mistakes that Briggs points out are all too common.
My partial solution would be a short course focussing on the ‘negative’.
-Don’t pay attention to wee ‘p-values’.
-Beware of smoothed (adjusted, fiddled, rearranged, cherry-picked) data.
-Ignore studies where a casual model is absent.
-Scorn studies that go ‘fishing’ for a model to fit ‘observations’.
-Avoid studies confusing correlation and causation.
And so on.
Pound the negatives into the students head in the same way that the ‘times’ tables were pounded into my head when young. 8X8=64. Piece of cake.
When in doubt, like my friend, call another department. In his case ‘urology’ in this case ‘statistics’.
A business, perhaps of interest to those running the Briggsian Universe.
“This is Matt, we are here to help. And please remember the first call is free!”
Always enjoy the comments and a great blog.
Pound the negatives into
An Apology:
Please mentally remove (from comment 11) the last words “Pound the negatives into” which represent a typo I missed.
Won’t happen again, he said, making a prediction.