Despite the best efforts of their wives, some married men remain recalcitrant and uncooperative. They refuse to or are incapable of taking their training seriously. Some of these men, who remain just on the pliant side of incorrigible, are talked into attending therapy or marriage counseling.
Men, incidentally, wouldn’t need this therapy and could avoid all marital strife if they followed the two simple rules known to the husbands in my family: (1) Do what you’re told, or when that becomes too distasteful or burdensome, (2) Feign deafness and hide.
It also helps to have at least two children so that when something is found to be put away in the wrong place, or when the bread bag is found to be improperly sealed, then you have somebody who can take the blame. You don’t need rigorous proof that whatever is wrong was not your fault, you just need a momentary distraction so that you can make your escape.
Anyway, for the men who haven’t learnt these lessons and for their wives, Laurie Abraham has written The Husbands and Wives Club: A Year in the Life of a Couples Therapy Group.
An excerpt of that book was recently on Slate: Can You Really Predict the Success of a Marriage in 15 Minutes? In it, Abraham look critically at the work of John Gottman, a “marriage researcher” who claims to be able to predict which couples will divorce within six years, with 80% accuracy.
Gottman was featured, every so lovingly, in Malcom Gladwell’s Blink. Gladwell tells how Gottman filmed several couples discussing, or arguing, some topic for as short as five to fifteen minutes. He used facial recognition techniques that he said could distinguish between coyness, genuine happiness, or combativeness. He also coded the speech content into various categories.
He then fed all this information into a computer—is there nothing these miraculous devices cannot do!—and out popped a prediction whether or not the couple filmed would divorce. He then wrote up his results, went through peer review—peer review!—and found his fame after Gladwell found him.
He also told good stories. Vivid, with plenty of detail, like a novelist. Hot tip: a rattling good yarn is a necessity if you want your theory to gain acceptance.
Abraham, however, was suspicious of Gottman. Were his fancy statistical models better than just guessing? Gottman claimed to be able to fit his model like a glove to old data. But Abraham knew that what was “absolutely required by the scientific method…is to apply your equation to a fresh sample to see whether it actually works.” This, Gottman never did.
She continues (I wept with delight when I read this):
The fundamental problem is that no matter how many equations, even quite similar ones, Gottman generates, we have no real idea of his forecasting power because of the way he reports his data. In statistics, you can’t judge the predictive oomph of anything without knowing the population prevalence of the event or condition you’re studying…Gottman talks about his equation’s “accuracy rates,” but scientists typically don’t use such language. They report false-positive and false-negative rates and then use those figures with prevalence to ascertain the effectiveness of whatever test or method is at issue.
Abraham gives an example of how Gottman’s claims fare, which I’ll modify a bit. In 1998, 160 out of every 1000 couples divorced (that’s the base rate). That means 840 did not divorce. I now introduce my patented Briggs’s Divorce Predictionator™: it says that, for every couple that comes to me, they will not divorce. I repeat: my prediction for any couple is that they stay married. I will be right 840 / 1000 = 84% of the time. Correct?
Gottman should be able to beat this prediction if his model is any good. That is, if his facial-recognition-speech-encoding-Gladwell-impressing model can’t beat 84%, then he has no skill. We already saw that Gottman claimed only 80% accuracy, so his model is in deep kimchi.
It has no skill. Skill is a formal statistical concept which we can formally calculate. For base rates less than 50%—ours is 160 / 1000 = 16%—the formula is:
Skill = (True positives – False positives) / (True positives + False negatives).
In Gottman’s case, “True positives” are those divorces he correctly predicts. “False positives” are those couples he said will divorce but don’t. And “False negatives” are those couples he said will remain married but that actually divorce. In order for a forecast to be skillful, skill must be greater than zero, or True positives must be larger than False positives (skill is bounded by +1 and -infinity).
We don’t know Gottman’s false positives and negatives because he never reveals them. But we do know that since his accuracy rate is less than Briggs’s Divorce Predictionator™, it must be that his skill is negative.
And so we have another in an already enormous, and ever growing, fund of beautiful models that are worthless.
Update Harvery also sends along this article, The Hazards of Predicting Divorce Without Crossvalidation, from the Journal of Marriage and Family. Worth a browse for the technically minded reader.
Update Thanks to Chuck for noticing the typo in the skill formula.