William M. Briggs

Statistician to the Stars!

Page 394 of 571

Manzi: What Social Science Does—and Doesn’t—Know

This article is nothing but an extended link to a must-read piece in City Journal. Internet still once daily. Thanks to reader I. for suggesting this topic.

If you haven’t already, you must read Jim Manzi’s City Journal article, What Social Science Does—and Doesn’t—Know: Our scientific ignorance of the human condition remains profound.

This man is my brother. Except for the common mistake in describing the powers of “randomization”, Manzi expounds a view with which regular readers will be in agreement:

[I]t is certain that we do not have anything remotely approaching a scientific understanding of human society. And the methods of experimental social science are not close to providing one within the foreseeable future.

His article will—I hope—dampen the spirits of the most ardent sociologist, economist, or clinician. For example, I cannot think of a better way of describing our uncertainty in the outcome of any experiment on humans than this:

In medicine, for example, what we really know from a given clinical trial is that this particular list of patients who received this exact treatment delivered in these specific clinics on these dates by these doctors had these outcomes, as compared with a specific control group. But when we want to use the trial’s results to guide future action, we must generalize them into a reliable predictive rule for as-yet-unseen situations. Even if the experiment was correctly executed, how do we know that our generalization is correct?

Amen and amen.

Manzi did a sort of meta analysis, in which he examined the outcomes of 122 sociologist-driven experiments. Twenty-percent of these had “statistically significant” outcomes; that is, had p-values that were publishable, meaning they were less than the magic 0.05 level.

Only four of the twenty percent were replicated and none provided joy. That is, there were no more magic p-values.

This is the problem with classical statistics: it lets in too much riff raff, wolves in statistically significant clothing. It is far too easy to claim “statistical significance.”

Classical statistics has the idea of “Type I” and “Type II” errors. These names were not chosen for their memorability. Anyway, they have something to do with the decisions you make about the p-values (which I’ll assume you know how to calculate).

Suppose you have a non-publishable p-value, i.e., one that is (dismally) above the acceptable level required by a journal editor. You would then, in the tangled syntax of frequentism, “fail to reject the null hypothesis.” (Never, thanks to Popper and Fisher, will you “accept” it!)

The “null hypothesis” is a statement which equates one or more of the parameters of the probability models of the observable responses in the different groups (For Manzi, an experimental and control group).

Now, you could “fail to reject” the hypothesis that they are equal when you should have rejected it; that is, when they truly are unequal. That’s the “Type II” error. Re-read this small section until that sinks in.

But you could also see a publishable p-value—joy!—“by chance”. That is, merely because of good luck (for your publishing prospects), a small p-value comes trippingly out of your statistical software. This is when you declare “statistical significance.”

However, just because you see a small p-value does not mean that null hypothesis is false. It could be true and yet you incorrectly reject it. When you do, this is a Type I error.

Theory says that these Type I errors should come at you at the rate at which you set the highest allowable p-value, which is everywhere 0.05. That is, on average, 1 in every 20 experiments will be declared a success falsely.

Manzi found that the 122 experiments represented about 40 “program concepts”, and of these, only 22 had more than one trial. And only one of these had repeated success: “nuisance abatement”, i.e. the “broken windows” theory. Which, it must be added, hardly needed experimental proof, its truth being known to anybody who is a parent.

The problem, as I have said, is that statistical “significance” is such a weak criterion of success that practically any experiment can claim it. Statistical software is now so easy to use that only a lazy person cannot find a small p-value somewhere in his data.

The solution is that there is no solution: there will always be uncertainty. But we can do a better job quantifying uncertainty by stating our models in terms of their predictive ability, and not their success in fitting data.

This is echoed in Manzi:

How do we know that our physical theories concerning the wing are true? In the end, not because of equations on blackboards or compelling speeches by famous physicists but because airplanes stay up.


11 Comments

Everybody Has A Mental Disease: Diagnostic and Statistical Manual 5

I am so far Up North that I heard a radio interview with Uncle Ted Nugnet on the best kinds of arrowheads to bring down feral pigs. Internet still only once daily mlh


Step into my parlor, and let me wave my diagnosticulator at you. OK, let me just consult the book.

Ah! Just as I suspected. Since you yelled at that IRS agent during your audit, we know you suffer from temper dysregulation disorder with dysphoria. This is normally seen in children, and is what we used to call a temper tantrum. Actually, it is a mental disease.

When seen in adults such as yourself, it requires medication, if not confinement. It’s for your own good.

And speaking of children, you have some, do you not? With a guardian such as yourself already known to be suffering from a mental disease, your children are at risk. In fact, I’m going to write a prescription for Psychosis Risk Syndrome. You just give them these pills, OK?

The fifth edition of the Diagnostic and Statistical Manual of Mental Disorders, the DSM-V is on its way. Just as did DSM-II, DSM-III, and DSM-IV, the fifth entry in this best seller from the American Psychiatric Association will expand the number, kinds, and ranges of mental disease.

Thus, the APA will have fulfilled at least one of its functions: providing job advocacy for its members.

As in previous editions, we can predict that the creators of the maladies will be as certain sure that the diseases they discover will be just as real as the old diseases they discovered and said were real, only to later say that they were not.

For example, homosexuality, which used to be a mental disease classified by the DSM, is now normal. In DSM-IV, “problems with law enforcement” was a disease, and in number five, it won’t be. Why? Because of “cultural issues”. Cancer is cancer whether you are Chinese of Zimbawayan. But a mental disease can depend on where you live?

Anyway, now “heartache over a lost spouse” will be a disease “Suck it up, Bob. Everybody dies. What made your wife special? Walk it off before they come at you with the nets.”

According to an official draft which was released in February, “internet addiction” nearly made the cut. But the consensus committee to develop a consensus—all science is done by Consensus nowadays; didn’t you know?—thought they couldn’t quite get away with this one.

Still, it was too sexy to ignore. Reports are that it will be relegated to the Appendix. As if that will stop shrinks and social workers from making the diagnoses.

Even if they cannot (yet) officially use “internet addiction”, they can still take somebody’s iPhone from them, and if those deprived squabble, they can be hit with “miscellaneous discontinuation syndrome.”

Is it merely your malcontented author complaining, or is there something untoward occurring in the ranks of the white-coated lithium prescribers? After all, we know more about the brain and its functioning now then we knew when the last manual appeared in 1994. Shouldn’t these advances be incorporated?

Professor Til Wykes from Kings College London coauthored an editorial in the August issue of the Journal of Mental Health arguing that DSM-V is “leaking into normality. It is shrinking the pool of what is normal to a puddle.”

Docs at Psychiatric Times, social workers at New York University, and others are are writing in to say, “Slow Down.”

We can visualize the so-called progress of the DSM with this cartoon, which shows the range of human behavior.

At the extremes, people are considered, or actually are, nuts. But as time progresses, what used to be considered eccentric, is viewed as abnormal and in need of medicating, or at least worthy of employing members of the APA to treat.

Nobody, not even the APA, disagrees that, according to the progressive editions of the DSM, a narrower and narrower range of behavior had been labeled “normal.”

If this trend towards limiting acceptable behaviors continues, psychitary will soon be indistinguishable from psychoanalysis or scientology, systems in which it is believed everybody suffers and must be “made clear”, and where only occult experts can guide one on the path to enlightenment.


17 Comments

Correlation and Causality: Have A Drink, Get Rich

I am down to one brief internet connection daily, found, intermittently, at a coffee shop in town. I apologize for lack of or slowness in answering questions.

According to the estimable Gallup organization, rich people drink more than poor ones. And since most of us want to be rich—I do—then statistics says that we should learn to properly cock an elbow.

This is true: folks who make less than twenty grand are shockingly abstemious; less than half of them know that a toast isn’t just something that is buttered. By the time median pay levels are reached, thirty to fifty thou., two out of every three have learned the art of the sip.

But it isn’t until we reach the upper echelons—seventy-five big ones or better—that we encounter regularly gins and tonics, Campari, rocks, and the Glenlivet.

It cannot be a coincidence that the more education one receives, the more one is likely understand the proper mechanics of the cocktail shaker and the true meaning of straining a drink. This is because education and income are, as everybody knows, correlated.

Diplomas are not always causative of dollars, however. A degree does not guarantee dinero in every instance. Those of us who are unspeakably learned tend—there is no other word—to be able to support Congresspersons in the manner to which they have been accustomed, but not always.

Albert Einstein, no mental slouch, did not die obscenely wealthy. Neither did Galileo, Mozart, or Mark Twain. And I am under no burdens of noblesse oblige myself.

Yet Ambassador Sean Penn, Bruce Springsteen, those earnest ladies from The View, and miscellaneous quarterbacks and some other athletes, are all bursting with the stuff.

From these observations we know that education and intelligence do not cause wealth; that is, we know that ignorance does not cause penury. All it takes is one instance of a intelligent poor person, or a rich politician to know that intelligence and wealth are not causatively related for all people.

Though they might be in some, or for others at some times. For example, many firms stipulate that one must have this-and-such degree or a demonstrated level of ability for their employees salaries to attain certain levels. In these cases, intelligence—always loosely defined—is causatively associated with income.

As long as there are no exceptions: let just one in—we discover one person with the proper credentials yet who has not been given his due—and we move from causation to correlation.

But causation is still, in the realm of the everyday world, a matter of probability. For example, we push a button and see that television comes on. Pushing that button is what caused the TV to turn on, we say. But it could be—who knows?—that every time thus far we have pushed that button the TV has coincidentally turned on, perhaps because of mysterious vibrations as yet unclassified by physics. That is, it could be that the clicker (or remote, if you will), has nothing to do with the television’s status.

As David Hume said, it is only because we become accustomed to things happening, one after the other, that we develop the notion of one thing causing another. Be clear: when I say that causation is probabilistic, I do not mean that causation is not present: something caused the events we witness to occur.

Instead, it is our knowledge of causation that is probabilistic. Now, we can develop a theory, which is a device that says that if certain things are true, then these events definitely occur. However, theories only pushes uncertainty back one level, because our knowledge of the conditions is never perfectly sure.

For example, there is a long history in physics of us guessing the causes of events, only to discover something deeper is at work. Yet because we have done so well making accurate, or tolerably approximate predictions, with our physical (semi) causative theories, our belief in the causative principles of theories is strong.

This is not so for human relations. As a prominent sociologist once said of measurements of behavior, “Everything is correlated with everything else.” This being so, we should expect sociologists, politicians and the like, to temper their theoretical judgments, to acknowledge vast uncertainty.

Thus, the political theory that by pushing this button, all or most people will do this, cannot and should not carry much conviction.

At least, not in an person intelligent enough to distinguish between correlation and causation. To believe strongly in any sociological theory is thus no different than reasoning that the best time to buy a lottery ticket, is right after having a drink.


9 Comments

Three Cards: One Black, One Red, One Half & Half

Just a short one today, all. I’m heading further North. So far from civilization that when I was a boy, it was a cause to celebrate when a cit thirty miles distant opened a Holiday Inn with an—get this— enclosed swimming pool. Such a marvel!

Good probability problems are those that affront the intuition, and the best are simple-sounding but cause fights or furious indignation. The Monty Hall dilemma is one such problem.

Most have heard of the stink made when the self-named Marilyn Vos Savant submitted her correct solution to Parade magazine. Not just civilians, but your actual college mathematics professors wrote in to say Wrong! But wrong they were.

We met something of the same symptom with the “One Boy Born Tuesday” problem. How the hell can knowing which day a kid was born give us any information about his sex?!

This kind of thing doesn’t fit into any of the ordinary slots we set aside for probability knowledge. And that’s what makes it such a wonderful example, because it forces us to carve out new slots once understanding arrives.

The “Three Cards: One Black, One Red, One Half & Half”, and the subject of today’s puzzler, is nothing more than a gentle push on the boundaries. The trick will come in trying to expand it by adding a twist, something I’ll invite you to do.

I have three cards, one of which is all black, one all red, and the last black one side and red the other. Before you on the table is a card whose face is red. Given this information—for we recall that all probability is conditional on stated premises—what is the chance that the other side is black?

Most intuitions suggest that the chance is 1/2, since you have either the all red card or the card that is half and half. But the actual chance is 1/3. Here’s why.

In all probability problems, the only thing to do is to lay out everything that can happen. This can be laborious, but once you do so you cannot go wrong. The mistake people make is to try and jump to the right answer using just their intuition.

What’s everything that can happen? Well, every card has two sides, 1 and 2. You could have seen B1 or B2 of the black card, R1 or R2 of the red card, or CR (combo red) or CB (combo black). You see red, and that can happen three ways: R1, R2, or CR.

If you see R1, then the other side must be R2. Likewise, if you see R2, the other side must be R1. The third and final possibility is you see CR, which makes the other side CB. Thus, the probability—give all the evidence we have—is 1/3.

Not so bad, right? Almost to the point of being dull.

So can we think of a clever way of tarting it up? My guess is that adding seemingly irrelevant, but actually pertinent, information can add the necessary amount of bogglement. Like in the “One Boy Born Tuesday” problem.

How about this? Same initial premise of three colored cards, but this time I say I deal you the card on Tuesday. Given this additional information, what is the probability the other side is black?

I haven’t worked this out—it’s a spur-of-the-moment thought. It might well be that the day of the week is irrelevant. But can we prove it is so?


9 Comments
« Older posts Newer posts »

© 2015 William M. Briggs

Theme by Anders NorenUp ↑