From reader Paul comes this important question (I eliminated personal information to keep the reader’s anonymity):
I’m reading through your book, “Uncertainty” and while I enjoy the philosophical approach to probability, the book’s message, if i’m interpreting it correctly, is causing concern.
Especially, ch. 10.5 “Quantifying the Unquantifiable”. Are you suggesting that most of the field of applied psychology (and our reliance on surveys to assess unquantifiable things like personality, emotion, learning agility and so on) are all BS? Do you think all psychometrics is fallacious?
Most importantly, what do you recommend as the alternative? Say I need to develop a psychometric test to help organizations select “better qualified” candidates – how would this be accomplished without tests/batteries of tests that are, in some way, shape or form, attempting to measure the unquantifiable?
If you can’t tell, the tone of this message is of genuine curiosity mixed with a sinking feeling that organizational psychology may have fundamental flaws. And, oh, the replication crisis is also distressing. Thank you for your time.
I say that forced quantifications of the inherently unquantifiable produces vast over-certainty. Hence Uncertainty, of which I wish there were more. Quoting myself:
Nobody disputes that there are levels of, say, happiness. One can be amused, pleased, gratified, sated, satisfied, gleeful, ecstatic, serene, gloomy, depressed, sad, grieved, aggrieved, and on and on. Yet it is only hubris that allows a researcher to say, “How happy are you on a scale from 1 to 10?” and think he has well quantified this complex emotion merely because some people checked off a number. But even this might be okay, this crude, blundering quantifying of the unquantifiable—after all, this is the purpose of all those different words we have for happiness—except that the research must go on and submit his answers to classical statistical analysis. Calamitous over-certainty is the result.
Not only can you be amused and pleased, you can be (quoting Moby Thesaurus) “comfortable, composed, congruous, content, contented, convenient, convincing, correct, crapulent, crapulous, dancing, decent, decorous, delighted, desirable, dizzy, dovetailing, drenched, drunk, drunken, easy, easygoing, ecstatic, effective, effectual, efficacious, efficient, elated, eupeptic, euphoric, exalted, exhilarated, expedient, exuberant, exultant, fair, far-gone, favorable, favoring, feasible, felicitous, fit, fitted, fitten, fitting, flushed, flushed with joy, flustered, fortuitous, fortunate, fou, fructuous, full, full of promise, gay, geared, genial, genteel, giddy” and so on and on and on.
Each of these words has shades of meaning, with tendrils that reach and twist throughout our minds. There is a rich complexity and impenetrable depth to all our states of mind, of time and situation, of permanence and impermanence, the complexity of which can only be plumbed by poets and priests.
To think we could put happiness on a scale from (as I joke continuously) -17.4/e to sqrt(pi^pi) is nuts.
But since we can have different levels of happiness, people figure a numerical scale can work. “These numbers are close enough. Besides, if we don’t put numbers to these things, then we can’t analyze them numerically.” A true statement. Proving things have already gone wrong before the happiness “instrument” was invented.
This is not—do I need to repeat this not?—to say that psychology is impossible. It is perfectly possible, albeit it should be carried on with far, far less certainty.
Here’s a story. UHS’ stock price up more than 10% after $127M DOJ settlement news.
BuzzFeed began posting articles in December 2016 about UHS’ mistreatment of behavioral health patients, describing a pattern of involuntary admissions in cases where it allegedly was not justified to increase reimbursement.
These were psychiatric admissions. “Expected to keep beds full, former admissions workers from three UHS hospitals said they learned how to turn even passing statements that people made during assessments into something that sounded dangerous.”
Now Wikipedia, which does a fair summary on the Myers-Briggs (no known relation):
Although popular in the business sector, the MBTI exhibits significant scientific (psychometric) deficiencies, notably including poor validity (i.e. not measuring what it purports to measure, not having predictive power or not having items that can be generalized), poor reliability (giving different results for the same person on different occasions), measuring categories that are not independent (some dichotomous traits have been noted to correlate with each other), and not being comprehensive (due to missing neuroticism). The four scales used in the MBTI have some correlation with four of the Big Five personality traits, which are a more commonly accepted framework
You might have seen this article: “Paper Says Some Psychiatric Diagnoses ‘Scientifically Meaningless’“.
We need only mention the name of Freud, and recall the well known truth of how easy it is to manipulate professional organizations, such as the APA, to bow to the zeitgeist.
Okay, so what. Over-certainty is at pandemic levels, but decisions do have to be made.
It is not that testing cannot be helpful. Of course it can. I have an in-depth article on intelligence testing here: The Limitations And Usefulness Of IQ.
Potential firemen are asked to lift heavy weights and jog around with hoses to see if they are fit. They are also asked commonsense precise work-related questions, like “Should you throw gasoline or water on a fire to put it out?” but the scores of these are often said to be “racist” or “sexist” or whatever, so they can’t be used.
The danger is over-reliance on the scores of these tests when they can be used. It’s much harder and costlier to vet somebody the old-fashioned way of prolonged questions and putting them through the gauntlet. Certainly it is less quantifiable. About the replication crisis, see this page.
The goal people have is to develop an entirely objective “rubric” which anybody can use to rate a person along some dimension or dimensions. I argue that this is an impossibility. Not that it is unlikely, but it is impossible. Perfection, therefore, is not to be had. Therefore my conclusion remains the same: Be less certain.
To support this site and its wholly independent host using credit card or PayPal (in any amount) click here
A practical goal is the ordering of states and predicting of future actions (“buy my stuff more often” or “graduate from college”). The “number” is just a convenient symbol. Unfortunately, people tend to “reify” that, and treat it like a physics measure, particularly hard scientists/engineers and finance/accounting types. Often its like a young child or a dog. You point at something, say “look”, and they look at the pointer. Receiving an “ENTJ” or whatever on an MB is at best an ordinal, not a cardinal (an actual amount.)
The “replication crisis” is a measurement crisis, caused by bulls**t “measurement” techniques. Spending any time talking with a lot of users/psychological scientists and you’ll hear amazing ideas. Look at the study areas that replicate and those that don’t.
The more attention to mesurement and technique, the more likely it is to replicate.
One person rates his happiness as an 8; another, as a 4. Only a fool would suppose the former was twice as happy as the latter. We used to use ordinal scales to measure the cure of internal coating of beverage cans using the “blush test” and comparing the milkiness of the coating to a series of type examples labelled 0, 1/2, 1, 2, 3, 4, 5. The 1/2 was a nice touch. But it is against the law to calculate the numerical mean of what are essentially labels.
A practical application of Meyers-Briggs: it helped me develop 16 characters for the novel The Wreck of “The River of Stars”
Re “reliance on surveys to assess unquantifiable things”. I am an engineer and always ask what are the units of measurement. For instance, the unit of electromotive force is the Volt. The NIST has the definition of the Volt and the standard to measure the Volt. If I want to measure Volts I use a Voltmeter with calibration traceable to NIST. Most of the things which these psychologists purport to measure are simply abstractions which have no physical existence.
Physical performance tests are sexist, because they show differences between the sexes (and also races).
Mental performance tests are racist, because they show differences between the races (and also sexes).
Reality is therefore at fault, and must be banned by law.
This is your starting point for all investigations into “Why?”
A related problem to quantifying the unquantifiable is the problem of misunderstanding what you are measuring, and therefore having bad intuitions about causality.
For example, if you take a survey, asking people how happy they are, you are not–in any sense–measuring happiness; you are measuring how happy people report they are, which is a much different thing. So if you have some theory about happiness, say that happy people are more likely to help out a stranger, and you try to find a correlation between self-reports of happiness and helping strangers, you have two potentially huge sources of error: one is the fact that there really isn’t a number associated with happiness, so *any* potential correlation is questionable, and the other is that your number isn’t even associated with happiness; it’s associated with happiness self-reports, which may or may not itself be correlated with happiness.
Perhaps they ought to be measuring Joy, as one can have Joy in the Lord in the midst of sorrow, if one seeks Him with all of one’s heart.
Happiness is relative to so many things, especially to the view one takes or does not take of one’s own life, whether the view is unbiased or not.. One can be happy one minute and sad or discontented the next.
The mind and emotions are powerful states of consciousness (thank-you dictionary.com), the emotions, an affective state of consciousness in which joy, sorrow, fear, hate, or the like, is experienced, as distinguished from cognitive and volitional states of consciousness.
So, away we go, one day this way and the next day, that way. But when consciously grounded in Christ, we are no longer tossed about as St. James says, but as we stay confident in Him, He brings us through.
Bring on the tests, but refine them to the truth of human beings, at times swaying with the wind.
God bless, C-Marie