The Ease of Cheating With Statistics

Thanks to readers Ari Schwartz and Tom Pollard for suggesting this article.

Take any two sets of numbers, where the only restriction is that a reasonable chunk inside each set has to be different than one another. That is, we don’t want all the numbers inside a set to be equal to one another. We also want the sets to be, more or less, different, though it’s fine to have some matches. Make sure there is a least a dozen or two numbers in each set: for ease, each set should be the same size.

You could collect numbers like this in under two minutes. Just note the calories in an “Serving size” for a dozen different packages of food in your cupboard. That’s the first set. For the second, I don’t know, write down the total page counts from a dozen books (don’t count these! just look at the last page number and write that down).

All set? Type these into any spreadsheet in two columns. Label the first column “Outcome” and label the second column “Theory.” It doesn’t matter which is which. If you’re too lazy to go to the cupboard, just type a jumble of numbers by placing your fingers over the number keys and closing your eyes: however, this will make trouble for you later.

The next step is trickier, but painless for anybody who has had at least one course in “Applied Statistics.” You have to migrate your data from that spreadsheet so that it’s inside some statistical software. Any software will do.

OK so far? You now have to model “Outcome” as a function of “Theory.” Try linear regression first. What you’re after is a small p-value (less than the publishable 0.05) for the coefficient on “Theory.” Don’t worry if this doesn’t make sense to you, or if you don’t understand regression. All software is set up to flag small p-values.

If you find one—a small p-value, that is—then begin to write your scientific paper. It will be titled “Theory is associated with Outcome.” But you have to substitute “Theory” and “Outcome” with suitable scientific-sounding names based on the numbers you observed. The advantage of going to the cupboard instead of just typing numbers is now obvious.

For our example, “Outcome” is easy: “Calorie content”, but “Theory” is harder. How about “Literary attention span”? Longer books, after all, require a longer attention span.

Thus, if you find a publishable p-value, your title will read “Literary attention span is associated with diet”. If you know more about regression and can read the coefficient on “Theory”, then you might be cleverer and entitle your piece, “Lower literary attention spans associated with high caloric diets.” (It might be “Higher” attention spans if the regression coefficient is positive.)

That sounds plausible, does it not? It’s suitably scolding, too, just as we like our medical papers to be. We don’t want to hear about how gene X’s activity is modified in the presence of protein Y, we want admonishment! And we can deliver it with my method.

If you find a small p-value, all you have to do is to think up a Just-So story based on the numbers you have collected, and academic success is guaranteed. After your article is published, write a grant to explore the “issue” more deeply. For example, we haven’t even begun to look for racial disparities (the latest fad) in literary and body heft. You’re on your way!

But that only works if you find a small p-value. What if you don’t? Do not despair! Just because you didn’t find one with regression, does not mean you can’t find one in another way. The beauty of classical statistics is that it was designed to produce success. You can find a small, publishable p-value in any set of data using ingenuity and elbow grease.

For a start, who said we had to use linear regression? Try a non-parametric test like the Mann-Whitney or Wilcoxon, or some other permutation test. Try non-linear regression like a neural net. Try MARS or some other kind of smoothing. There are dozens of tests available.

If none of those work, then try dichotomizing your data. Start with “Theory”: call all the page counts larger than some number “large”, and all those smaller “small.” Then go back and re-try all the tests you tried before. If that still doesn’t give satisfaction, un-dichotomize “Theory” and dichotomize “Outcome” in the same way. Now, a whole new world of classification methods awaits! There’s logistic regression, quadratic discrimination, and on and on and on… And I haven’t even told you about adding more numbers or adding more columns! Those tricks guarantee small p-values.

In short, if you do not find a publishable p-value with your set of data, then you just aren’t trying hard enough.

Don’t believe just me. Here’s an article over at Ars Technica called “We’re so good at medical studies that most of them are wrong” that says the same thing. A statistician named Moolgavkar said “that two models can be fed an identical dataset, and still produce a different answer.”

The article says, “Moolgavkar also made a forceful argument that journal editors and reviewers needed to hold studies to a minimal standard of biological plausibility.” That’s a good start, but if we’re clever in our wording, we could convince an editor that a book length and calories correlation is biologically plausible.

The real solution? As always, prediction and replication. About which, we can talk another time.