Posts filed under 'Bad graphics'
Yesterday’s post was entitled, “You cannot measure a mean”, which is both true and false depending—thanks to Bill Clinton for the never-ending stream of satire—on what the meaning of mean means.
The plot I used was a numerical average at each point. This implies that at each year there were several direct measures that were averaged together and then plotted. This numerical average is called, among other things, a mean.
In this sense of the word, a mean is obviously observable, and so yesterday’s title was false. You can see a mean, they do exist in the world, they are just (possibly weighted) functions of other observable data. We can obviously make predictions of average values, too.
However, there is another sense of the word mean that is used as a technical concept in statistics, and an unfortunate sense, one that leads to confusion. I was hoping some people would call me on this, and some of you did, which makes me very proud.
The technical sense of mean is as an expected value, which is a probabilistic concept, and is itself another poorly chosen term, for you often never expect, and cannot even see, an expected value. A stock example is a throw of a die, which has an expected value of 3.5.
Yesterday’s model B was this
B: y = a + b*t + OS
I now have to explain what I passed over yesterday, the OS. Recall that OS stood for “Other Stuff”; it consisted of mystery numbers we had to add to the straight line so that model B reproduced the observed data. We never know what OS is in advance, so we call it random. Since we quantify our uncertainty in the unknown using probability, we assign a probability distribution to OS.
For lots of reasons (not all of them creditable), the distribution is nearly always a normal (the bell-shaped curve), which itself has two unobservable parameters, typically labeled μ and σ^2. We set μ=0 and guess σ^2. Doing this implies—via some simple math which I’ll skip—that the unknown observed data is itself described by a normal distribution, with two parameters μ = a + b*t and the same σ^2 that OS has.
Unfortunately, that μ parameter is often called “the mean“. It is, however, just a parameter, an unobservable index used for the normal distribution. As I stressed yesterday (as I always stress), this “mean” cannot be seen or measured or experienced. It is a mathematical crutch used to help in the real work of explaining what we really want to know: how to quantify our uncertainty in the observables.
You cannot forecast this “mean” either, and you don’t need any math to prove this. The parameter μ is just some fixed number, after all, so any “forecast” for it would just say what that value is. Like I said yesterday, even if you knew the exact value of μ you still do not know the value of future observables, because OS is always unknown (or random).
We usually do not know the value of μ exactly. It is unknown—and here we depart the world of classical statistics where statements like I am about to make are taboo—or “random”, so we have to quantify our uncertainty in its value, which we do using a probability distribution. We take some data and modify this probability distribution to sharpen our knowledge of μ. We then present this sharpened information and consider ourselves done (these were the blue dashed lines on the plot yesterday).
The unfortunate thing is that the bulk of statistics was developed to make more and more precise statements about μ : how to avoid bias in its measurement, what happens (actually, what never can happen) when we take an infinite amount of data, how estimates of it are ruled by the central limit theorem, and on and on. All good, quality mathematics, but mostly besides the point. Why? Again, because even if we knew the value of μ we still do not know the value of future observables. And because people tend to confuse their certainty in μ with their certainty in the observables, which as we saw yesterday, usually leads to vast overconfidence.
From now on, I will not make the mistake of calling a parameter a “mean”, and you won’t either.
March 10th, 2008
There’s something of a controversy whether photographer Roger Fenton placed cannon balls in a road and then took pictures of them. He also took a picture of the same road cleared of cannon balls. Apparently, there is a question whether the cannon balls were ON the road when he got there, or possibly they were OFF and he placed them there to get a more dramatic photo. This drama unfolds at Errol Morris’s New York Times blog.
Whether they were first ON or OFF (Morris uses the capitals letters, so I will, too), excited considerable interest, with hundreds of people commenting one way or the other, each commenter offering some evidence to support his position.
Some people used the number (Morris uses the ‘#’ symbol) and position of the balls, others argued sun shadows, some had some words about gravity, and so on. Morris compiled the evidence used by both sides, ON (cannon balls on first) and OFF (cannon balls placed there by Fenton), and he presented this summary picture (go to his blog to see the full-sized image):
This is an awful graph: the order of evidence types is arbitrary, it would have been better to list them in order of importance; the use of color is overwhelming and difficult to follow; and, worst of all, the two graphs are on an absolute scale. 288 people supported ON, and 153 OFF, so counting the absolute numbers and comparing them, as this picture does, is not fair. Of course the ON side, with almost twice as many people, will have higher counts in most of the bins. What’s needed is a percentage comparison.
One of his blog’s readers provided just such a comparison, in the following pictures (again, go to his blog to see the full-sized images):
These are, of course, pie charts, and while they are slightly better than the original bar charts, they are still awful. Three simple things wrong are again the order of evidence is arbitrary, the are an overwhelming number of colors, and the they needlessly print the exact percentages—down to the 10th place! Once again, there is a larger sin: The main purpose of these pictures is to compare the different percentages supporting each type of evidence, but to do that your eye must jump from picture to picture, find the relevant slice, and then go back to the original to check for the difference. This makes the reader work very hard to get the information.
I drew this:
The dark blue lines indicate that a larger percentage of ON people supported that type of evidence; dark red lines indicate that a larger percentage of OFF people supported that type of evidence. The types of evidence labels appear on the side with the larger percentage. The types of evidence are also ordered by importance.
For types of evidence Topography/Climate, Camera/Exposure, Cannon ball properties, Gravity and Physics, there is little difference between the two groups. But those who supported ON, thought Number and position of balls, Sun shadows, and Practical concerns were much more important. Those who supported OFF, thought Character (of the photographer)/Artistic, Shelling, and placement of Rocks were much more important.
I’m not certain how interesting, or relevant, any of this data is, but what is important to us is that graphs can find the interesting and relevant data, as long as you are willing to put in the effort to create good ones. Nearly always, the default graphs available in packages like Excel, fall short of the mark (yes, a very, very weak cannonball pun).
Oh, Morris guesses that OFF is the truth.
December 28th, 2007
If you think it’s good news that the death rates by stroke, heart disease, cancer, and just plain accidents have declined last year (diabetes deaths have remained steady), then you’re not trying hard enough to find the dark lining to this silver cloud.
But, thank goodness, death rates from COPD have “skyrocketed“, so we don’t need to stop worrying! The New York Times even supplied a graph (below) as evidence of this calamity.
There are two things wrong with this bleak outlook. The first is an error in logic, the second is one of bad graphics.
Can you see what’s wrong with the statistical graph? Looks like a dramatic increase in COPD deaths, right? Well, maybe. But hasn’t the population, for men and women, also increased—skyrocketed—since 1980? I have only been able to discover (from this site) the COPD deaths per 100,000 up until 2004 (not 2005 like the Times picture), but here is that picture:
They certainly increased a bit for women, stayed steady for men, and even leveled off around 2000 for women. No skyrockets here. This leads to a simple graphics rule: always normalize by the base population!
To investigate the logical error, first read this, which is advice doctors are supposed to give patients on hearing the allegedly cheering news that death rates have fallen (advice is from this site):
Explain to patients that while age-adjusted death rates are declining, heart disease, stroke, cancer, accidents, COPD and diabetes remain the leading causes of death
Ready? First, that list of things that can kill you together caused nearly all deaths. Now, it is an observational fact that these diseases, heart disease, stroke, accidents, and cancer are killing people at a slower rate, and that diabetes kills at the same rate. It is also true that people are dying each year. And something must kill these people. Therefore, since COPD is the only major killer left in the list, death rates due to it must increase, even if the behaviors and external causes that lead to COPD remain unchanged.
The corollary to this is that you have to die of something, and that some disease will have to be the leading cause of death. No matter how much money you spend, or how many laws that you pass, or how much you exercise and diet, some malady will always be Number One in the Grim Repear’s play book.
December 17th, 2007
This graphic comes from the New York Times article “Social Security Disability Cases Last Longer as Backlog Rises.” It obviously intends to show how applications have increased since 1998.
This is a terrible plot.
The reason is not that you should never, with only rare exceptions, use a bar chart. They are simple to construct, but there are nearly always better alternatives.
But the evil of bar charts is well known. The reason this plot is bad has to do with the number 0. Notice that the chart starts at 0, even though it isn’t until 2 million or so that we meet our first number. The only reason that the chart starts with 0 is that it is true that you can’t have less than 0 applications. This is not a good reason. They should have started with a higher number.
Don’t think it makes a difference? Then take a look at this re-drawing:
I drew the same plot, but started at 2 million, a more reasonable number, and I also eliminated the distracting bars. Notice anything new?
It’s the same exact numbers, but now we see a remarkable spike in 2002, after which the number of new applications actually settles down to a steady 2.5 million or so.
Why is that spike there? I have no idea—I am not an economist—but then the New York Times reporter also doesn’t explain the huge increase. For the simple reason that he never saw it—you cannot see the spike using the bar chart. You can see it using the standard, non-sexy, scatter plot (of course, an artist at the paper could have a go at the re-drawing and make it much prettier).
There are three other plots in the New York Times’s article; each are bar plots, and each are bad in the same way. Each can be re-drawn like the one above, and each has a surprise in it. It’s an illuminating exercise that I leave to the reader.
I leave you with this question: what good stories are you missing from using bad graphics?
December 10th, 2007
