Over-Optimism In Physician Prognoses

Poor Michael Boren lay dead at one of the ugliest places in New York City, the on-ramp to the Queensboro Bridge. Heart attack. He was only 51. He made it through just two boroughs of Sunday’s Five Boro Bike Tour before ceasing to be (materially).

Thing is, Boren’s doctor “gave him the OK to participate in the race.”

Another busted forecast.

Or rather, another plastered prognosis. Turns out doctors, like most of us, aren’t such great predictors. Human behavior is too complex for anybody to nail with anything approaching consistent accuracy. This includes experts prognosticating in their fields of expertise, too. That was the lesson of, among many others, Phil Tetlock’s Expert Political Judgment.

I was reminded of this when reading a physician’s lamentation of over-confidence, in which he pointed to a British Medical Journal paper “Extent and determinants of error in doctors’ prognoses in terminally ill patients: prospective cohort study.”

This is just one example of an endless supply. The study: “343 doctors provided survival estimates for 468 terminally ill patients at the time of hospice referral.” There are a lot of words in the paper, but it all comes down to this picture, which isn’t as good as it could be:

The chart is backwards to custom, which would place the predicted survival days on the “x” or horizontal axis, and the observed data on the “y” or vertical axis. The chart is also on a log-log scale, which makes it difficult to appreciate the magnitude of the errors.

But, forgiving all that, let’s take a look. If the doctors gave perfect forecasts, all the dots would line up on the solid black diagonal line: the distance from this line is the error. Not too many dots on or near the line.

Put your finger in the leftmost dot at 30 days, which is one doctor’s prediction of how long his patient would survive. Drop down from that 30 to the x-axis to learn that the patient actually lived just 2 days. That’s a huge error, especially considering this is the End Of The Road, a time when families are making hard decisions.

Because of the log-log scale the errors are larger than you would think in some places. For example, look at topmost dot at just over 1000 days, which is about three years. The patient only lived a month (30 days). That’s a bigger error than the one at the far left, where the doctor said the patient would live around 400 days, but where the patient made it only to the next day (oops). The error is smaller here even though the distance to the black line is longer, because the scale is not linear.

Notice that most of the dots are on the north side of the line which means, for this group of patients and doctors, the forecasts were too much on the optimistic side; that is, the doctors said patients would live a lot longer than they actually did. You can also see a bit of cultural bias in the data: e.g., the cluster of points predicting 90 days (3 months) to live.

One problem in this study is the discrete nature of the prognoses. No doctor and no patient believes he will live precisely 90 days when given that forecast. There is some plus-or-minus which is understood, but maybe not in the same way by both parties. The doc’s window may be narrower than the patient’s, or vice versa.

Every good forecast provides an indication of its uncertainty. A prediction of “90 days plus or minus two months” is different from one which says “90 days to a year.” And of course, doctors more often give predictions in this form. The uncertainty is needed because the decisions a patient and his family makes given a forecast are vastly different than the decisions the doctor makes.

Incidentally, assessing the quality of predictions which come with uncertainty is more difficult than making simple plots like this, but the methods to do so are well understood.

And there’s more to think of. Should a physician give his patients hope by telling them they’ll live longer than he really thinks? “Buck up Mr Jones! I’ve known patients in your condition who lived for years.” Optimism is a sort of placebo, is it not? But can you tell a patient he will live “years” when you believe that patient is circling the drain? Optimism has limits, and the power of the mind (placebo effect) not omnipotent. Bad forecasts aren’t helpful to families, either.