From reader Paul comes this important question (I eliminated personal information to keep the reader’s anonymity):
I’m reading through your book, “Uncertainty” and while I enjoy the philosophical approach to probability, the book’s message, if i’m interpreting it correctly, is causing concern.
Especially, ch. 10.5 “Quantifying the Unquantifiable”. Are you suggesting that most of the field of applied psychology (and our reliance on surveys to assess unquantifiable things like personality, emotion, learning agility and so on) are all BS? Do you think all psychometrics is fallacious?
Most importantly, what do you recommend as the alternative? Say I need to develop a psychometric test to help organizations select “better qualified” candidates – how would this be accomplished without tests/batteries of tests that are, in some way, shape or form, attempting to measure the unquantifiable?
If you can’t tell, the tone of this message is of genuine curiosity mixed with a sinking feeling that organizational psychology may have fundamental flaws. And, oh, the replication crisis is also distressing. Thank you for your time.
I say that forced quantifications of the inherently unquantifiable produces vast over-certainty. Hence Uncertainty, of which I wish there were more. Quoting myself:
Nobody disputes that there are levels of, say, happiness. One can be amused, pleased, gratified, sated, satisfied, gleeful, ecstatic, serene, gloomy, depressed, sad, grieved, aggrieved, and on and on. Yet it is only hubris that allows a researcher to say, “How happy are you on a scale from 1 to 10?” and think he has well quantified this complex emotion merely because some people checked off a number. But even this might be okay, this crude, blundering quantifying of the unquantifiable—after all, this is the purpose of all those different words we have for happiness—except that the research must go on and submit his answers to classical statistical analysis. Calamitous over-certainty is the result.
Not only can you be amused and pleased, you can be (quoting Moby Thesaurus) “comfortable, composed, congruous, content, contented, convenient, convincing, correct, crapulent, crapulous, dancing, decent, decorous, delighted, desirable, dizzy, dovetailing, drenched, drunk, drunken, easy, easygoing, ecstatic, effective, effectual, efficacious, efficient, elated, eupeptic, euphoric, exalted, exhilarated, expedient, exuberant, exultant, fair, far-gone, favorable, favoring, feasible, felicitous, fit, fitted, fitten, fitting, flushed, flushed with joy, flustered, fortuitous, fortunate, fou, fructuous, full, full of promise, gay, geared, genial, genteel, giddy” and so on and on and on.
Each of these words has shades of meaning, with tendrils that reach and twist throughout our minds. There is a rich complexity and impenetrable depth to all our states of mind, of time and situation, of permanence and impermanence, the complexity of which can only be plumbed by poets and priests.
To think we could put happiness on a scale from (as I joke continuously) -17.4/e to sqrt(pi^pi) is nuts.
But since we can have different levels of happiness, people figure a numerical scale can work. “These numbers are close enough. Besides, if we don’t put numbers to these things, then we can’t analyze them numerically.” A true statement. Proving things have already gone wrong before the happiness “instrument” was invented.
This is not—do I need to repeat this not?—to say that psychology is impossible. It is perfectly possible, albeit it should be carried on with far, far less certainty.
Here’s a story. UHS’ stock price up more than 10% after $127M DOJ settlement news.
BuzzFeed began posting articles in December 2016 about UHS’ mistreatment of behavioral health patients, describing a pattern of involuntary admissions in cases where it allegedly was not justified to increase reimbursement.
These were psychiatric admissions. “Expected to keep beds full, former admissions workers from three UHS hospitals said they learned how to turn even passing statements that people made during assessments into something that sounded dangerous.”
Now Wikipedia, which does a fair summary on the Myers-Briggs (no known relation):
Although popular in the business sector, the MBTI exhibits significant scientific (psychometric) deficiencies, notably including poor validity (i.e. not measuring what it purports to measure, not having predictive power or not having items that can be generalized), poor reliability (giving different results for the same person on different occasions), measuring categories that are not independent (some dichotomous traits have been noted to correlate with each other), and not being comprehensive (due to missing neuroticism). The four scales used in the MBTI have some correlation with four of the Big Five personality traits, which are a more commonly accepted framework
You might have seen this article: “Paper Says Some Psychiatric Diagnoses ‘Scientifically Meaningless’“.
We need only mention the name of Freud, and recall the well known truth of how easy it is to manipulate professional organizations, such as the APA, to bow to the zeitgeist.
Okay, so what. Over-certainty is at pandemic levels, but decisions do have to be made.
It is not that testing cannot be helpful. Of course it can. I have an in-depth article on intelligence testing here: The Limitations And Usefulness Of IQ.
Potential firemen are asked to lift heavy weights and jog around with hoses to see if they are fit. They are also asked commonsense precise work-related questions, like “Should you throw gasoline or water on a fire to put it out?” but the scores of these are often said to be “racist” or “sexist” or whatever, so they can’t be used.
The danger is over-reliance on the scores of these tests when they can be used. It’s much harder and costlier to vet somebody the old-fashioned way of prolonged questions and putting them through the gauntlet. Certainly it is less quantifiable. About the replication crisis, see this page.
The goal people have is to develop an entirely objective “rubric” which anybody can use to rate a person along some dimension or dimensions. I argue that this is an impossibility. Not that it is unlikely, but it is impossible. Perfection, therefore, is not to be had. Therefore my conclusion remains the same: Be less certain.
To support this site and its wholly independent host using credit card or PayPal (in any amount) click here