Despite the best efforts of their wives, some married men remain recalcitrant and uncooperative. They refuse to or are incapable of taking their training seriously. Some of these men, who remain just on the pliant side of incorrigible, are talked into attending therapy or marriage counseling.
Men, incidentally, wouldn’t need this therapy and could avoid all marital strife if they followed the two simple rules known to the husbands in my family: (1) Do what you’re told, or when that becomes too distasteful or burdensome, (2) Feign deafness and hide.
It also helps to have at least two children so that when something is found to be put away in the wrong place, or when the bread bag is found to be improperly sealed, then you have somebody who can take the blame. You don’t need rigorous proof that whatever is wrong was not your fault, you just need a momentary distraction so that you can make your escape.
Anyway, for the men who haven’t learnt these lessons and for their wives, Laurie Abraham has written The Husbands and Wives Club: A Year in the Life of a Couples Therapy Group.
An excerpt of that book was recently on Slate: Can You Really Predict the Success of a Marriage in 15 Minutes? In it, Abraham look critically at the work of John Gottman, a “marriage researcher” who claims to be able to predict which couples will divorce within six years, with 80% accuracy.
Gottman was featured, every so lovingly, in Malcom Gladwell’s Blink. Gladwell tells how Gottman filmed several couples discussing, or arguing, some topic for as short as five to fifteen minutes. He used facial recognition techniques that he said could distinguish between coyness, genuine happiness, or combativeness. He also coded the speech content into various categories.
He then fed all this information into a computer—is there nothing these miraculous devices cannot do!—and out popped a prediction whether or not the couple filmed would divorce. He then wrote up his results, went through peer review—peer review!—and found his fame after Gladwell found him.
He also told good stories. Vivid, with plenty of detail, like a novelist. Hot tip: a rattling good yarn is a necessity if you want your theory to gain acceptance.
Abraham, however, was suspicious of Gottman. Were his fancy statistical models better than just guessing? Gottman claimed to be able to fit his model like a glove to old data. But Abraham knew that what was “absolutely required by the scientific method…is to apply your equation to a fresh sample to see whether it actually works.” This, Gottman never did.
She continues (I wept with delight when I read this):
The fundamental problem is that no matter how many equations, even quite similar ones, Gottman generates, we have no real idea of his forecasting power because of the way he reports his data. In statistics, you can’t judge the predictive oomph of anything without knowing the population prevalence of the event or condition you’re studying…Gottman talks about his equation’s “accuracy rates,” but scientists typically don’t use such language. They report false-positive and false-negative rates and then use those figures with prevalence to ascertain the effectiveness of whatever test or method is at issue.
Abraham gives an example of how Gottman’s claims fare, which I’ll modify a bit. In 1998, 160 out of every 1000 couples divorced (that’s the base rate). That means 840 did not divorce. I now introduce my patented Briggs’s Divorce Predictionator™: it says that, for every couple that comes to me, they will not divorce. I repeat: my prediction for any couple is that they stay married. I will be right 840 / 1000 = 84% of the time. Correct?
Gottman should be able to beat this prediction if his model is any good. That is, if his facial-recognition-speech-encoding-Gladwell-impressing model can’t beat 84%, then he has no skill. We already saw that Gottman claimed only 80% accuracy, so his model is in deep kimchi.
It has no skill. Skill is a formal statistical concept which we can formally calculate. For base rates less than 50%—ours is 160 / 1000 = 16%—the formula is:
Skill = (True positives – False positives) / (True positives + False negatives).
In Gottman’s case, “True positives” are those divorces he correctly predicts. “False positives” are those couples he said will divorce but don’t. And “False negatives” are those couples he said will remain married but that actually divorce. In order for a forecast to be skillful, skill must be greater than zero, or True positives must be larger than False positives (skill is bounded by +1 and -infinity).
We don’t know Gottman’s false positives and negatives because he never reveals them. But we do know that since his accuracy rate is less than Briggs’s Divorce Predictionator™, it must be that his skill is negative.
And so we have another in an already enormous, and ever growing, fund of beautiful models that are worthless.
———————————–
Thanks to Harvey Motulsky, author of Intuitive Biostatistics, who suggested this article.
Update Harvery also sends along this article, The Hazards of Predicting Divorce Without Crossvalidation, from the Journal of Marriage and Family. Worth a browse for the technically minded reader.
Update Thanks to Chuck for noticing the typo in the skill formula.
I just read the Gottman section in Gladwell’s ‘Blink’ and was jaw-dropped-impressed. I thought Gottman, from his description, was a creepy, love-less, magician you don’t want over for dinner. But still, I bought it.
Looking back on it, Gottman also laid out the 4 flaws to a relationship, or (what you should have called this post ) 4 things that make relationships fail. The worst one being, “Contempt.”
I bought that too.
But now it seems a lot like a Cosmo article with numbers, a computer, and a video camera.
‘ it says that, for every couple that comes to me, they will not divorce.’
What was that phrase again?
‘Self selecting sample group’.
That was it.
I suspect the Briggs Divorce Predictionator might not hit those dizzy 84% heights?
I think there is something wrong with your Skill formula – it doesn’t match the bounds that you quote. It appears to be a sign error.
Or maybe my brain just isn’t working this morning 😉
Chuck,
No, you’re brain is fine. It’s mine that’s defective. I’ve fixed the typo. Thanks.
With respect, I would imagine that the divorce rate for couples going to see Gottman is not equal to the population as a whole. In fact, I’d guess it’s higher.
I can’t imagine a more horrible occupation than marriage counselor. Despite the skill of the essentially braindead model, you don’t want to get involved, Briggs. Really you don’t.
Gottman was driven mad. He couldn’t stand more than 15 minutes with a squabbling couple. I suggest 5 minutes is too much. Let the pros handle this, like Dr. Phil, who has also lost his marbles. Try gardening. It doesn’t pay as much as marriage counseling, but you get to keep your marbles.
“He then fed all this information into a computer—is there nothing these miraculous devices cannot do!”
Everybody that has followed the AGW debate knows that computers are able to foretell the future so they should be able to tell if a marriage will fail. In the old days if we wanted to know the future we had to consult the haruspex or Karnak the Magnificent. Now we just use the computer.
Briggs:
Well worthless in the sense of being useful to others, true, but quite valuable to Gottman’s marketability in the feel-good faux-scientific milieu which keeps him supplied with BMW’s and Perrier water. Or did until Abraham came along.
Is the divorce rate quoted per year (1998) or some other period of time? If per year it seems rather high. The original claim is for six years after counselling of this select group. Without this information I cannot understand your analysis.
Look, the chickens are going to get their necks wrung and get eaten anyway. So why not examine their entrails?
I’m confused. Isn’t the population here simply all those couples that are enrolled in therapy? As opposed to all couples, where the 84 % statistic was presumeably derived? If so, the divorce rate would surely be higher than 16%.
All,
Quite right that the base rate should be of the population projected. I had seen in Blink, and in Abraham’s article, more general claims that our man could predict any couple’s divorce; apparently that’s how Abraham took it, because I used her numbers . Also take a look at the article referenced in the Update: it contains more general criticisms.
Wonder if Gottman got his idea from this.
Dr. Strangelove: “Well, that would not be necessary, Mr. President. It could easily be accomplished with a computer. And a computer could be set and programmed to accept factors from youth, health, sexual fertility, intelligence, and a cross-section of necessary skills. Of course, it would be absolutely vital that our top government and military men be included to foster and impart the required principles of leadership and tradition.”
Amazing what those computers can do. Better to be a politician than marriage counselor.
Matt:
Something still feels wrong about the formula since it seems to automatically allow a high skill rating when the event occurs significantly more (or less )frequently than 50% of the time. Surely the skill is defined by the excess above chance given the marginals? Otherwise simply guessing that no couples would be divorced produces a high skill rating.
Bernie,
This formula is only when the event is rare; i.e., when it occurs less than or equal to 50% of the time. If the event is frequent—more than 50% of the time; perhaps Gott’s quarrelsome clients—the formula changes to this:
(True Negatives – False Negatives) / (True Negatives + False Positives)
The article from the Journal of Marriage and Family is worth reading. From Slate
If this is the case, the reported “80% accuracy†is a measure of fit of the model to the data. A misinterpretation or confusion between “goodness-of-fit measure†and “predictive accuracyâ€.
The computer is innocent! I don’t think “we’re yet evolved to the point where we’re clever enough to handle a complex a situation†as solving an optimization problem by hand. And seriously, I am not saying that humans are too stupid to do so; see this. ^_^
I wouldn’t say the model is worthless because I still have no idea what it is, and it seems to me that the model simply hasn’t been verified or tested properly. Furthermore, failure is part of the process, and I personally don’t think any failure is worthless. No?
Briggs,
Do you have a reference for the skill formula (and, maybe, a derivation?). I see that it is a penalized accuracy, but have never seen it referred to as a skill formula.
bill r,
Take a look at my Resume tab, scroll down to my list of papers, and look at those with the word “skill” in the title. The early Biometrics is the main reference.
“I will be right 840 / 1000 = 84% of the time. Correct?” Well, sure, if we are only talking about one year. But the 80% claim is for 6 years.
If we assume the divorce rate stays the same at 16% each year, then we have:
Year 1: 1000 * 0.16 = 160 couples divorce. 840 couples remain.
Year 2: 840 * 0.16 = 134 couples divorce. 840 – 134 = 706 couples remain.
Year 3: 706 * 0.16 = 113 couples divorce. 706 – 113 = 593 couples remain.
Year 4: 593 * 0.16 = 95 couples divorce. 593 – 95 = 498 couples remain.
Year 5: 498 * 0.16 = 80 couples divorce. 498 – 80 = 418 couples remain.
Year 6: 418 * 0.16 = 67 couples divorce. 351 couples remain.
So, at the end of 6 years the patented Briggs’s Divorce Predictionatorâ„¢ has a 35.1% success rate. That’s a lot less than 80%.
* there are a lot of assumptions here that probably aren’t true. For example, divorce rates probably vary by how long the marriage has lasted so far.
** I’m not claiming to be statistically rigorous here. For example, I haven’t been as strict as possible with rounding, etc. But the basic point still holds.
*** I still agree with you that Gottman hasn’t shown he can predict divorces. I just think your example misses the mark.
Todd P,
You went wrong in a very interesting way.
Over whatever period of time—3 years, 6 years, 7, whatever—suppose there are 1000 divorces. If the divorce rate is 160 per 1000 over this time period—the statistic I was assuming—then 160 couples will have divorced and I will have been right 840 times.
Your calculation is more of an insurance against divorce. Think of it this way: with your method, by some year (I’ll you figure it) all couples will have divorced. Not too likely.
Yes, as I said, the assumptions are probably untrue. Divorce rate changes over time.
That doesn’t impact the main point: You can’t compare your 1-year prediction to his 6-year prediction (basing the results on a single year’s divorce statistics.) Apples != oranges.
Year 1: 1000 * 0.16 = 160 couples divorce. 840 couples remain.
Year 2: 840 * 0.16 = 134 couples divorce. 840 – 134 = 706 couples remain.
Year 3: 706 * 0.16 = 113 couples divorce. 706 – 113 = 593 couples remain.
The error in the calculation/model is that you assume that the divorced couples don’t then remarry a different partner and then get divorced. What caused the first divorce is probably likely to cause the following divorces.