MRI Claims 92% Accuracy Predicting Sexual “Orientation”. My Schmengee Mediscan 3000™ Beats It

The news, considered marvelous to some because it seems to confirm how good MRIs are at uncovering secrets of the brain, is that MRIs can predict “sexual orientation” with 92% accuracy.

The peer-reviewed paper is “Accurate machine learning prediction of sexual orientation based on brain morphology and intrinsic functional connectivity” in Cerebral Cortex by Benjamin Clemens and others.

They announce “We found an average accuracy of 62% (±6.72) for predicting sexual orientation based on [brain gray matter volume] and an average predictive accuracy of 92% (±9.89) using [resting-state functional connectivity].”

I, too, giggled at that “+ 9.89”, but let that pass. For now.

In any case, this 92% sounds like a big number. Is it?

I don’t have an MRI, but I do have the Schmengee Mediscan 3000™ (the deluxe chromium model). It is a special form of scanner that, like MRI, can judge people’s “sexual orientation”. The fantastic thing about it is that you don’t have to come to the scanner. It comes to you—remotely.

So good is the Schmengee Mediscan 3000™ that it doesn’t even have to know who it’s scanning. In fact, it’s even easier than this. It simply registers “Normal” for every scan.

That is, no matter who the Schmengee Mediscan 3000™ scans, it always and forever says “Normal”. For everybody.

Without reading further—do try this; don’t peek—what accuracy do you think the Schmengee Mediscan 3000™ scanner would have at guessing “sexual orientation”?

No peeking.

Did you peek?

The accuracy would be about 93%. Which is greater, of course, than 92%, which means our Schmengee Mediscan 3000™ scanner is better than the MRI.

It’s around 93% if Gallup’s survey is correct, that is. They say “LGBT Identification in U.S. Ticks Up to 7.1%“. If 7% are abnormal, that means 93% or so are normal. Which means a scanner that predicts normal for everybody would be accurate 93% of the time. It would only be wrong for those who “identify” as abnormal.

Given these facts, which machine would you rather use? The MRI or Schmengee Mediscan 3000™?

Why would you want to use an MRI when the Schmengee is better? Because the MRI is more expensive? Because it has all sorts of sciency theory behind it? Because it is wielded by Experts? Because even though it’s not as accurate it’s guesses are more scientific, in the sense the predictions it nailed were because of science, while those that it missed are because of science not yet perfected?

Now there is more to it than this. Unfortunately, I do not have the paper, but only the Abstract. (I hear the Sci-hub lady is on the run; they don’t have current papers anymore.) I did discover a popular article that referenced this study. That article said the MRI study correctly identified all 45 normals and 82.9% of 41 self-described homosexuals. That’s 92% overall. This is all we need.

Two points, the first easy.

This 92% appears to be the in-sample accuracy. How the model fit the data, that is. These are not independent predictions of data never seen or used before.

As regular readers should know by now it is possible to create a model that fits to whatever accuracy one desires. They could have got 100% if they wanted. This is always possible. No matter the kind of data. Which means we can’t trust any model just because, and only because it fits data well.

This means announcing accuracy of fit is not exactly worthless, but nearly. You will recall that the entire practice of classical statistics is centered around announcing model fit. P-values and Bayes factors, for instance. Which means the practice of classical statistics is ____________ (you fill in the blank).

Second, let’s take the in-sample accuracy statistics as a given, and pretend that they will apply to everybody. This will not be true. That is, the real-life accuracy of this MRI will be very much lower than 92% in practice (my guess is ~70–80%). But let’s suppose in practice it always guesses all normals correctly and all abnormals with 83% accuracy.

In any large sample we’ll have about 0.93 x N normals and 0.07 x N abnormals. That means the accuracy of predictions in large samples will be around (0.93 x N x 1 + 0.07 x N x 0.83) / N = 98.8% accuracy.

If that’s so, and the MRI really could reach that accuracy, then since 99% > 93% of the Schmengee, then the MRI model should be preferred.

But like I said, there is no chance the out-of-sample performance will be as good as the in-sample. Suppose the out-of-sample performance is 95% as good as the in-sample performance. This is incredibly generous, as any builder of models knows. Like I said parenthetically, something more like 70% is likely, or less. But suppose our scientists are much brighter than normal, and give them 95%.

Then the out-of-sample performance will be something like (0.93 x.95 + 0.07 x 0.83 x.95) = 93%.

In other words, the MRI model in real life is expected to be just as good as the Schmengee, and likely a lot worse. Which means we should prefer the Schmengee.

And which also means the MRI claims of being able to identify sexual “orientation” are likely not true.

What we have done today is called a skill test. It should always be applied to statistical models. It almost never is.

Late Addendum: It is not that is impossible MRIs might be able to identify “orientation”. Take broken legs, an injury. Machines can identify who has had one skillfully, but perhaps not perfectly, because the injury leaves traces. Only here the evidence is far from conclusive.

Subscribe or donate to support this site and its wholly independent host using credit card click here. Or use the paid subscription at Substack. Cash App: $WilliamMBriggs. For Zelle, use my email: matt@wmbriggs.com, and please include yours so I know who to thank.