Skip to content
October 26, 2008 | 8 Comments

Anybody see this one?

The book is The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives by Deirdre Nansen McCloskey and Steve Ziliak.

From the description at Amazon:

The Cult of Statistical Significance shows, field by field, how “statistical significance,” a technique that dominates many sciences, has been a huge mistake. The authors find that researchers in a broad spectrum of fields, from agronomy to zoology, employ “testing” that doesn’t test and “estimating” that doesn’t estimate. The facts will startle the outside reader: how could a group of brilliant scientists wander so far from scientific magnitudes? This study will encourage scientists who want to know how to get the statistical sciences back on track and fulfill their quantitative promise. The book shows for the first time how wide the disaster is, and how bad for science, and it traces the problem to its historical, sociological, and philosophical roots.

This is part of the theme I’ve long been pushing. McCloskey and Steve Ziliak are shocked, perplexed, and bewildered that classical statistics and p-values are still being used.

I’m not so shocked. They want people to abandon p-values and start using effect sizes. A fine first step, but one that doesn’t solve the whole problem.

I say we should drop p-values like Obama dropped Rev. Wright, eschew effect sizes like Joe Biden did reality, and return to observables. Let me, as they say, illustrate with a (condensed) example from by book.

Suppose there are two advertising campaigns A and B for widget sales. Since we don’t know how many sales will happen under A or B, we quantify our uncertainty in this number using a probability distribution. We’ll use a normal, since everybody else does, but the example works for any probability distribution.

Now, a normal distribution requires two unobservable numbers, called parameters, to be specified so that you can use it. The names of these two parameters are μ and σ. Both ad campaigns need their own, so we have μA and σA, and μB and σB. Current practice more or less ignore the σA and σB, so we will too.

Here is what “statistical significance” is all about.

Actual sales data under the two campaigns A and B is taken. A statistic is calculated: Call it T. It is a function of differences in the observed sales under both campaign. Never mind how it’s calculated. T is not unique, and for any problem dozens are available. With T in hand, the classical statistician makes this mathematical statement:


and then the infamous p-value is calculated, which is

   Probability(Another T > Our T given that μAB)

where the “Another T” is the statistic we would get if we were to repeat the entire experiment again. Do we repeat it again? No, so we are already in deep waters. But never mind.

If the p-value is less than the magic number of 0.05, then the results are said to be statistically significant.

Quick readers will have spotted the major difficulty. What does equating two unobservable parameters in order to calculate some weird probability have to do with whether the campaigns are different than one another?

The words are not much, which is why McCloskey and Ziliak call the dependence on p-values a cult.

They recommend, in its place, estimating the effect size, which is this:

   μA – μB.

Eh. It’s part way there, but it’s still a statement about unobservable parameters (and it still ignores the other unobservable parameters σA and σB).

What people really want to know is this:

   Probability(Sales A > Sales B given old data).

Or they’d like to estimate the actual sales under A or B. There are new ways that can calculate these actual probabilities of interest. However, you won’t learn these methods in any but the most esoteric statistics class.

And that is what should change.

Because, I am here to tell you, you can have a p-value as small as you like, you can have an effect size as big as you like, but it can still be the case that

   Probability(Sales A > Sales B given old data) ~ 50%!

which is the same as just guessing. Yes, the actual, observable numbers, the real-life stuff, the physical, measurable, tangible decisionable reality can be no different at all. At least, we might not be able to tell they are any different.

And that’s the point. The old ways of doing things were set up to make things too certain.

I wouldn’t go so far as to say reliance on the old ways was cultish. Most people just don’t know of the alternatives.

October 24, 2008 | 20 Comments

Health care crisis!?

Take a look at his picture:

Life expectancy rates through time

This is from an article by “The Numbers Guy” Carl Bialik at the Wall Street Journal. The story is about how life expectancy calculators are not terribly accurate. This really isn’t much of a surprise, but the picture should be.

This is because both presidential candidates, and of course many other people, nervously claim that there is a “Health Care Crisis! We have to do something!”

Yes, it’s so bad that the people are living longer and longer and longer… This picture says that whatever the crisis is, it clearly doesn’t have to do with that part of health that keeps people alive. I would argue that that part is the most important; apparently, others disagree.

This is another example of the phenomenon that the better things get, the more people complain. Or maybe people don’t complain more, they complain at the same rate, but because things are better, the complaints are about matters and points that are increasingly trivial.

Hasn’t somebody given a name for this dynamic?


Uncle Ted’s Manifesto: Ted, White, & Blue: a review

Uncle Ted

Read Ted, White, & Blue: The Nugent Manifesto. You will not agree with everything, some or lot of it might even infuriate you, but you not regret the time spent.

If you don’t know much about Nugent other than he’s a rock musician, let me tell you something you might not believe. Uncle Ted can write, and write well. His prose crackles with the same energy as his music. The words on the page flow; they practically jump off and get in your face. They force you to think, especially if you don’t agree with him. You can’t just say (to yourself), “He’s wrong.” He makes you say why. Doesn’t sound like much, but it’s no small trick.

Unlike just about every other celebrity over the last forty years, Uncle Ted lives clean and sober and always has.

Punks used to laugh at me, said how can you rock and not get high? Well I just stood my ground, and I watched those assholes fall and die. Cuz I just wanna go huntin, it makes me feel so good. I just gotta go huntin, try to find me in the woods.

Nugent is the ultimate environmentalist, a staunch steward of wildlife, an ardent advocate of hunter’s rights, and of the rights of the rest of us to own guns. This book isn’t about these things, but his previous book, God, Guns, & Rock and Roll, is. So if you’re looking for tips about the best kind of arrow head to bring down a buck, or you want to know why Fred Bear was a hero to so many, buy that book and not this one.

Ted, White & Blue really is a manifesto. Here are some of his non-standard proposals:

  • Refuse to fund healthcare for people who don’t care about their health…
  • Eliminate all welfare except temporary benefits for military personal and their families. Able-bodied Americans who refuse to work will be sent to Cuba, Mexico, England, and France.
  • Create a $100,000 reward for any U.S. citizen who shoots and kills a paroled felon during an assault or home invasion.
  • Eliminate the IRS, institute a national sales tax, and force the U.S. government to live within a budget tied to actual revenues.
  • Pass a constitutional amendment limiting citizen employment in the federal, state, or local government jobs to 5 percent of the U.S. workforce.
  • Remove and open all levees and dams in New Orleans and make people live on high ground. Give no more handouts to cover stupid mistakes of any kind.
  • Make it illegal to sue any business simply for the criminal misuse of their legal product.
  • Encourage all states to expeditiously execute all convicted child molesters.

I love the constitutional amendment idea.

Uncle Ted smiles

God gave man a soul; a powerful, instinctual moral and intellectual True North compass that completely differentiates us from all other living creatures….It is soulless to forbid a good citizen the right to carry a gun for self-protection while you dare to actually charge that citizen (subject) to pay for your armed security detail, Ted Kennedy.

Nugent is one of the few—left or right—willing to say things like this: “If you are pissed off about where you see the country going, remember this: you are to blame.” And this “Caring about a problem doesn’t solve a damn thing.” Raising awareness anyone? “[S]ome goofy Americans somehow they are entitled to these material creature comforts even though their wages cannot support this non-essential junk.” “Nobody owes you a thing. Nothing. Zilch. Zero. Nada. Everything you will get out of life will be based solely on what you put into it…You will get no more than what you are willing to bust your ass to earn.”

He is not a Republican nor a Democrat. His definition of televangelist is religious pimp. He isn’t anti-religion: “Let us all pray for good bombing weather.”

Uncle Ted made an huge, enormous mistake when he was a young man. He purposely got himself out of the Vietnam draft. He knows he must answer for this moral crime. I am tempted to say, a la the media, this “youthful indiscretion”, but I won’t, because what he did was wrong. The only question now is: can we forgive him?

Regrettably, I did not serve in the military upon graduation from high school. For that I am truly sorry…I admit to self-imposed, near-total insulation from worldly truth and the reality of Vietnam…This is no excuse for my woeful and deep disconnect from the critical events of the world–and I don’t offer it as one—but it is the truth…In order to provide some sort of restitution for my youthful disconnect, I have done what I can over the years for members of the armed forces…My enlightenment, though slow in coming, eventually arrived.

Four of my uncles served during the Vietnam war; my dad and another uncle were out of the Navy and Army before the war got going. I naturally served long after, but all my senior NCOs and officers were veterans1. I know how much it means to have people remember those who are serving. When I was in Okinawa, word had it the the folks at ESPN would mention our base on Sports Center one night. They did. A trivial thing, really. But everybody was very proud. Nugent’s concerts for the military in Iraq are deeply appreciated.

Some more Nugentisms. He does not want a fence built along our southern border, but would deport anybody who comes here illegally. “You don’t have a right to heath care. Got that? What you have is a personal responsibility for it.” “When was the last time you heard, read, or saw a story in the media that reported a citizen using a gun to stop a crime?” “Never forget you have a duty, not a right, to defend yourself and your family.”

Counting on ethanol to replace gasoline is akin to believing that rap is a legitimate musical genre, violent thugs deserve to be let out of their cages, or that animals have rights. Dogs chasing their tails make more sense than this bureaucratic KLSTRPHK. Ethanol is a joke.

Nugent has said that he is considering a return to Michigan to run for governor. Would you vote for him?

1None ever flipped out, went insane, became crazed, nor in other ways emulated Oliver Stone.