William M. Briggs

Statistician to the Stars!

Page 145 of 541

Poor Statistics Undermine The Reliability Of Neuroscience

I'm all aglow

I’m all aglow

Note carefully the picture which accompanies this post. The right-most glow is centered on the upper-middle-fifth amygdalic cingulatum region of your author’s brain. Statistics show that this region is “associated” with feelings of joy; more specifically, the shivers of delight one experiences when saying, “I told you so!”

(The other smaller glow to the left is “associated” with pleasant thoughts of Myers Dark rum, now ridiculously expensive.)

The synaptic juices started flowing and the glow glowed after I read “Many Neuroscience Studies May Be Based on Bad Statistics” in Wired, which opens:

The fields of psychology and cognitive neuroscience have had some rough sledding in recent years. The bumps have come from high-profile fraudsters, concerns about findings that can’t be replicated, and criticism from within the scientific ranks about shoddy statistics. A new study adds to these woes, suggesting that a wide range of neuroscience studies lack the statistical power to back up their findings.

The study is “Power failure: why small sample size undermines the reliability of neuroscience” in Nature Reviews Neuroscience by Katherine Button, John Ioannidis, and several others. Best thing about that paper was a short guide to terms researchers ought to know. My favorite:

Winner’s curse
The winner’s curse refers to the phenomenon whereby the ‘lucky’ scientist who makes a discovery is cursed by finding an inflated estimate of that effect. The winner’s curse occurs when thresholds, such as statistical significance, are used to determine the presence of an effect and is most severe when thresholds are stringent and studies are too small and thus have low power.

Button and team did a meta-meta analysis of fMRI studies and the like and discovered what will be no secret to regular readers: the statistics of these works ain’t too hot. Specifically, many (most?) have very low power. “The consequences of this include overestimates of effect size and low reproducibility of results.”

They looked at 48 meta-analyses, which comprised “730 individual primary studies.” The median power was 18%. If you don’t have a feel for that, the “normal” power for medical studies is 80%+. That’s the level grant granters want, anyway. Button’s finding means half the studies are worse than anemically powered.

The Scientist quotes Hal Pashler, a psychologist at the University of California, San Diego, as saying, “This paper should help by revealing exactly how bad things have gotten.” Can’t go too much by that, because it’s standard journalistic practice to fetch a quote from somebody who didn’t write the paper (and often didn’t read it). But in this case Pashler is right.

Or maybe I’m just happy to agree with him. Here’s why I do.

Point one, I did an extensive (maybe too extensive) critique of Sam Harris’s paper “The neural correlates of religious and nonreligious belief.” One of the worst papers, in a series of bad papers, that I’ve ever read. Shoddy experimental design, editorialism masked as science, data mysteriously disappearing, biases galore, et cetera.

If you’re only going to read two things, read these entries of the review “Can fMRI Predict Who Believes In God?” Part I and Part Last.

Points two and higher: click to one of these two reviews: Yet Another Study ‘Proves’ Liberal, Conservative Brain Differences, Brain Atrophy Responsible For Religious Belief?

I’ve done many more, but these capture the gist. Strange that these “studies” have a sort of theme to them, no?

Wired had the sense to ask why so many bad studies? Reason one: studies are too expensive. But since scientists must publish lest they perish, reason two: “[T]he pressure on scientists to publish often, preferably in high-profile journals, to advance their careers and win funding from the government.”

Since that pressure will not be lifted even after Button’s identification of systematic flaws, it is rational to expect a continuation of systematic flaws. Gives me a kind of job security, though.

IDing poor science doesn’t pay as well as generating it, however,. Actually it pays not at all. That’s why I think warm thoughts about rum: to keep my brain lit up.


——————————————————-

Thanks to Mike Flynn for pointing us to this fine news.

General Ethics: Peter Kreeft’s Summa Philosophica Part VII

Evil and Good, together again.

Evil and Good, together again.

Read Part VI.

Remember, we’re doing summaries of summaries here; only bare sketches are possible. Buy his book for more detail.

Question VII is fun: General Ethics. That morality really exists and such forth. Specific rules follow next time.

Article 1: Whether moral laws are objectively real?

Oh my yes. But isn’t it the case that “moral laws are not agreed to by everyone, but argued about, in all times and places”? And isn’t it true that moral laws are values and not facts? Or maybe morals are nothing but subjective feelings?

On the contrary, all men, in all times, places, and cultures, argue about whether certain acts…are morally right or wrong. But we argue only about objective truths, not subjective feelings…

Kreeft likes to say no one “argues about private, subjective feelings. No one responds to ‘I feel well’ with ‘No, you feel sick.'” Disagreement about objective matters does not prove subjectivity. That would make every scientific disagreement subjective. If you can find anybody who disagrees with any mathematical judgment, and if mere disagreement made topics subjective, then all of mathematics is subjective.

It’s true that “[a]ll judgments exist only in a subjective consciousness, including judgments about material things like size, judgments about mathematical truths, and judgments about moral laws. But the objects of these judgments, unlike feelings, are objective.”

We’re not empiricists, because if we were we’d have to toss out all of mathematics, logic, and all other metaphysical judgments like the existence of other minds. Positivism is dead.

Since there are objectively real moral laws, it is our task to discover them.

Article 2: Whether there are any universal, exceptionless norms?

Yes, afraid so.

Whenever an exception is made to a moral law, there is always a more general moral law that justifies the exception…The commandment forbids murder, not killing. The attempted murderer forfeits his right to life by threatening other lives.

Also, “[m]otives and situations cannot make an act that is wrong in itself to be right. They can only make an act that is otherwise right to be wrong. As T.S. Eliot said…about motives, ‘The last temptation is the greatest treason:/ to do the right thing for the wrong reason.'”

The clich´ says the exception proves the rule, but what it means is “that an exception presupposes a rule. Some rules have exceptions and some do not. If all rules have exceptions, what about that rule?”

Article 3: Whether ‘ought’ is a kind of ‘is’?

Yes. Yet Hume says an ‘is’ does not entail an ‘ought’ and Hume cannot be praise too much.

Now it is “common-sensical to argue that what you ought to be and do depends on what you are and on what is. For instance, ‘be a man’ is meaningless if addressed to an animal.” And it is only sensible to tell a man what to do what can be done. There cannot be a moral commandment to ride unicorns or to wipe the sauce dropped from flying spaghetti monsters.

“What is” is ambiguous. It can mean an empirical fact as distinct from a value…or it can mean, more broadly, any objective truth at all, including the truth that values are objective, and including truths about what these objective values are…

When we say justice “is” good, we are not merely using a logical copula as in “a unicorn is a mythical beast,” but we are asserting something about reality, about what really is.

Even better:

Moral arguments have two premises, not one. One of them is about values and one about facts. Thus there is no non sequitur. E.g. “Killing an innocent person is morally wrong; abortion is killing an innocent person; therefore abortion is morally wrong” derives the specific ethical conclusion from a more general ethical premise combined with a metaphysical premise about the real nature of abortion and its object.

Article 4: Whether there is a natural law?

Yes. And not in the biological sense of “We saw this group of men do this and that under such and such conditions.” While biology is important, men are not pandas or centipedes. From the anthropological observation that many men murder, we do not derive the moral law that murder is just. There’s more to us than our bodily workings.

Now anyone “can discover the natural law simply by an honest attention to his conscience.”

Since it is part of man’s essential nature to have free choice of will, rather than to be determined unfreely, the natural law for man is a law that prescribes rather than describes, or which describes what man ought to do rather than what man necessarily does.

Stating there is a natural law does not, of course, show us what the natural laws are. We do these next time.

Article 5: Whether evil is real?

Yes: don’t you read books? A popular non-argument is to say, “If we treat people as if they were good, they will respond by being good.” But this assumes people can respond oppositely, which is to say evilly, therefore there is evil.

Article 6: Whether evil is ignorance?

No. If you think so, try telling the cop who pulled you over you didn’t know the speed limit was only 35 MPH.

A brother fallacy of the preceding one is “Raising Awareness”, a radioactive byproduct of the Enlightenment which says that if only we expose people to “the truth” (which often just means “our belief”) they will automatically act properly or in a way which we desire. If this were true, every one of you reading this post would become saints (and so would your author). As much as I’d love that, smart money says it ain’t gonna happen.

Article 7: Whether mankind is insane to choose evil over good?

Yes, amen. Kreeft includes a third Enlightenment-driven counter-argument which claims that to broadcast we’re all bat-guano crazy is “harmful to our self-esteem, which is a necessary precondition to motivate us to improvement. Therefore it is counterproductive and should not be made.” There is no modern (non) crime worse than to hurt somebody’s feelings.

“When one is repeatedly given the opportunity for joy or misery and repeatedly chooses misery, it is not too severe to call this disorder insanity”. Now insanity has degrees; grips on reality or more and less firm. Those wearing sunglasses in the dark can see better than those with their heads wrapped in tinfoil covered in duct tape. So that just because we’re all nuts, doesn’t mean we can’t recognize people nuttier than us.

Article 8: Whether there is more good than evil in man?

Yes.

Evil cannot be greater than good because evil is a corruption of good and can exist only in a subject that is good by nature. If it corrupted all the good, it would cease to exist, since it would have no subject to exist in, like a parasite who killed its host.

As St Thomas Aquinas famously said, “Good can exist without evil, whereas evil cannot exist without good.”

Article 9: Whether virtue always brings happiness?

It does. Eventually. Though in the moments of sin, it feels so good. There is however always (as in always) a price.

[A]nyone can verify in his own experience, by experiment, so to speak, the fact that practicing virtue, especially charity, always produces deep happiness, while sinning against the virtues, especially charity, always produces deep discontent and inquietude.

Boethus: “All things seek again their proper courses and rejoice when they return to them.” And there’s no problem deriving pleasure from being virtuous.

It is not self-serving to desire one’s own happiness, or the “good of delight” (the bonum delictabile) together with the good of moral virtue (the bonum honestum), because they are consummated together. Proper self-love is not wrong but right, and proper self-love and proper altruism should coincide: thus “love your neighbor as yourself.”

Article 10: Whether sanctity is the key to ontology?

Yes—and hadn’t you thought about sanctity and ontology? Consider “Heidegger’s new and radical idea…that the mode of be-ing…[which] is usually called personhood or subjecthood, i.e. being an I rather than an It, is the key to being itself…and thus to ontology.”

The two realms of the personal subject and the ontological object, or person and being, are joined in God, who is the standard for both, and for their relationship. This when He revealed His own true, eternal name as “I AM,” He revealed the unity of personhood (“I”) and being (“AM”).

Also worthwhile is Kreeft’s speeach “Is Anything Really Right or Wrong?” (youtube video).

Read Part VIII.


Belief In God Associated With Psychiatric Symptoms

It's all cool, dude.

It’s all cool, dude.

Medicalizing Belief

Would you say that a guy who claims he believes in a “just” God suffers from “psychiatric symptoms”? What if we swap “just” for “critical.” Do we have a loon on our hand then or what?

Ross Pomeroy at Real Clear Science wants to believe it. And Nava Silton of the Department of Psychology at Marymount Manhattan College thinks she’s proved it. She has the statistics to back it up, gathered in the modern scientific way: by conducting a survey filled with loaded questions and then manipulating the answers mercilessly with tortuous statistics.

Silton wrote about it in the peer-reviewed “Beliefs About God and Mental Health Among American Adults” in the 10 April 2013 issue of the Journal of Religion and Health.

It’s Science!

Here’s what Silton did. Via Gallup, she asked 1,400-some folks a bunch of questions, some to assess their “psychiatric symptoms”, their religion, and their race.

What’s a “psychiatric symptom”? Answering high on questions like these (allowed range: 0-4). You suffered from “Generalized anxiety disorder” if you said you “Worried too much about different things”. The horror of “Social anxiety” is experienced by saying you “Became anxious doing things because people were watching”. You know you’re under the thumb of “Paranoia” if you “Felt that people were taking advantage of you.”

A doozy symptom is “Obsession”, which possessed you if you “Thought too much about things that would not bother other people.” What about “Compulsion”? That’s when you told the anonymous person on the phone, who was writing down all your answers, and who knew who you were and where you lived, that you “Repeated simple actions that realistically did not need to be repeated”.

“Beliefs about God were measured by participants’ responses to a list of adjectives describing God: absolute, critical, just, punishing, severe, or wrathful” on a scale from 1-4. A “Punitive God” score was created by adding answers to “punishing” and “wrathful.” A “Deistic God” added “absolute” and “just.” Finally, a “Benevolent God” added (in reverse) “critical” and “severe”.

Now the theological usefulness of these adjectives is best described as (and your author looked this up) vaporous. How much distance is there—exactly, now—between “punishing” and “just”? And what’s the specific difference between the opposite of “severe” (since that’s how the question was used, in its opposite sense) and “absolute”? How much time do you think respondents spent thinking of these questions? Did everybody interpret the adjectives in precisely the same way? The same as Silton?

What fooled Silton, and what bamboozles other “researchers” like her, is that her survey gave numerical results, which made it feel scientific. All worry about the meaning of the questions is gone and replaced by comforting manipulatable quantities.

Manipulate them she did, using several “regressions”, which are common statistical models. The wrong ones in situations like this, where the outcome has a limited range (regressions are designed for outcomes which are continuous and wide-ranging, the opposite of Silton’s “psychiatric symptoms”).

Results

Silton checked the “statistical significance” between the Benevolent God, Punitive God, and Deistic God scores with each of the five “psychiatric symptoms”. That makes 15 possibilities, of which she found wee p-values in only 8. The “discoveries” only pertained to Benevolent and Punitive God, with only Social anxiety, Paranoia, Obsession, and Compulsion.

For example, for every increase in the arbitrary Punitive God score of one unit, the curious Social anxiety “psychiatric symptom” increased on average 0.08 points. This is in the range statisticians classify as “trivial.” Worse—and here it gets technical—the explanatory power of this model measured by adjusted R22 was 0.04. This is less than trivial. What it means is that the increase in “symptoms” is only barely likely, that it only happens sometimes, at only a slightly higher rate than a coin flip, that decreases are just about as likely.

All Silton’s findings are like this: you have to squint to see them, and you must wear special glasses (provided at tenure ceremonies). Nevertheless, theory is on the line. The “Evolutionary Threat Assessment System Theory” to be specific. Belief in this theory was so strong in Silton that even with such watery results, she asserted “belief in a punitive God had a pernicious association with psychiatric symptoms”. There is hope, though: “belief in a benevolent God had a salubrious association” with the “psychiatric symptoms”.

This wasn’t her final word. No: future research is needed. Like “How might belief in a punitive God relate to depression and disordered eating?”


————————————————————————-

Thanks to Juan Ramirez for alerting us to this topic.

Two Kinds Of Bad Statistics Caused By Publish & Perish

Two (main) kinds of bad statistics: (1) where they are used to claim what is not true, and (2) where they are used to claim what is true.

The second might not seem bad, because a truth is trumpeted, and that is what one does with truths. But it is bad just the same, because the truths statistics “validates” did not need support. They were already well known to be true. Saying “Statistics has finally verified that X” where everybody already knew X has two consequences.

It starts you doubting your powers of reason, which did not need doubting. After all, they led you to X in the first place, and X is true. The second replaces the doubt with unwarranted trust in white-coated men. And that makes it much more likely that bad statistics of the first type are produced and accepted.

The reason bad statistics of the second type exist is that they must or the white-coated men will perish. As in “publish or perish.” There is a quota of papers required of all academics. These works may be good or bad, right or wrong, useful or harmful, penetrating or indifference, but regardless they must be written. Academics must write even when they don’t wish to, when they have nothing to say, when it would have been better had they remained quiet.

It is this absurd requirement which produces bad works of the second kind. (It has many other harmful effects, notably boosting the egos of the academics, but we can examine these another day.)

Example? How about “Vividness of the Future Self Predicts Delinquency”? A peer-reviewed work in Psychological Science by Jean-Louis van Gelder and others. Van Gelder opens the paper with these words:

The tendency to live in the here and now, and the failure to think through the delayed consequences of behavior, is one of the strongest individual-level correlates of delinquency.

This is our X, which everybody whose powers of reason are not stunted—as they are in, say, infants and those who have received overdoses of NPR—already knew. There was no reason in the world, save saving van Gelder’s job, that we had to be told this again, as if it were new or in doubt.

After all, it was from Aesop we learned the tale of the ant and the grasshopper, a story which was ancient before he wrote it down. Do you recall the crushing final line where Mr Ant, PhD (Assistant prof. up for tenure) lamented, “If only the grasshopper had a more vivid picture of his future self!”?

I had planned on reviewing the rest of the paper, which like all these things isn’t content with the obvious until it is lathered with dubious theory, but why bother. Did we really need to learn how a group of “young people” filled out sets of questionnaires from which wee p-values were squeezed? All to prove that those who thought about the future fared better in that unknown country than those who fiddled the day long? No, we did not.


« Older posts Newer posts »

© 2014 William M. Briggs

Theme by Anders NorenUp ↑