I used to review these papers in the hopes people would understand the old ways of doing statistics, in particular experiments on human subjects, were guaranteed to produce over-certainty.
I still want that. But now my co-equal, or even main, goal is for you to develop less respect for science. To not love any scientific announcement with the same ardor you might have done in the past. Science, though it sure is swell, is no longer the shining example of intellectual acuity it once was. There are still areas in science and technology free from the currents, but they are decreasing. Besides, too much love of science leads to scidolatry.
Anyway, our case study today is from Nature, “Effects of short-term exposure to particulate matter air pollution on cognitive performance” by Shehab and Pope.
PM 2.5, which is to say common dust, is all the rage in academia. Scientists like it because it’s a reliable paper producer; bureaucrats like it for its regulatory potential. “We must take a stand against dust! Consider the children!”
Ben Franklin, to name but one, was no intellectual slouch. He used to read books by candlelight. Many books, many candles. He read so much he had to invent bifocals. He went on to greater triumphs, as schoolchildren used to be taught. I’m sure they now learn he was racist, or whatever.
Point is that breathing in the dust created by burning all those candles didn’t seem to have any negative effects on his cognitive ability. Nor on the many others of his and earlier generations.
You would guess, if you were of the old-fashioned scientific bent, that by, say, 1850, after millennia of candle use, people would have noticed whether candles were making them stupider. Nobody did notice, and even noticed an opposite effect, such as allowing reading in dark rooms. Neither observation is definitive proof, but then, in science, most observations or experiments aren’t.
Smokers, too, inhale loads of particulate matter on purpose. They often report cognitive increases after partaking of pipe weed. Of course, these benefits come with other costs. And it’s probably not the dust, but the nicotine, that boosts synaptic flow.
Back to the paper. Shehab and Pope in one experiment gathered a group of 30 people, mostly students, and paid them to endure candle dust for an hour then gave them a questionnaire, a quiz. They did second experiment, which was much the same, but we can only endure so much, about going by a road.
The results from the MMSE test showed a statistically robust decline in cognitive function after exposure to both the candle burning and outdoor commuting compared to ambient indoor conditions…The findings from this study are potentially far reaching; they suggest that elevated PM pollution levels significantly affect short term cognition. This implies average human cognitive ability will vary from city to city and country to country as a function of PM air pollution exposure.
They gave 11 quizzes, which all gave numerical scores, once in normal air and again in air tinged with candle dust. They compared the scores from this pre and post.
It appears they report the mean of all the scores for the pre and then the mean of all the scores for the post. Which is the wrong thing to do. It should be the within-person change in score that is of interest. They did do paired t-tests to generate their wee Ps, which shows the understood this. But it appears they forget when tabulating the results. Maybe it’s men. My attention was flagging and I had just mainlined some beeswax.
Anyway, 11 quizzes, only one of which gave a wee P, and some of which showed higher (better) means for candle-dust exposure. Even if you love p-values, which shows a corruption of your soul, this is still depressingly underwhelming. If you adjust the P-values, as some do, for multiple tests, the “significance” here disappears.
But we remember that p-values grossly exaggerate evidence in favor of effects. The differences in means is so small, coupled with the noted standard deviations, allows us to guess that if these data were analyzed predictively, no effects at all would be noticed.
Did I mention the candles burned only for an hour?
This study suggests important implications for human cognitive ability and mental health and their dependence upon PM exposure. It suggests that citizens of more polluted cities and countries will have, on average, worse cognitive ability than they would have if air quality was better. Therefore, this study suggests that reductions in PM air pollution will not only result in improved human morbidity and mortality outcomes but also upon cognitive performance. Further work is now required…
Reminder: this breathless (get it? get it?) discussion was produced after just a handful of people were exposed to a candle for one hour, and where only one not-so-wee exaggerated P was discovered.
“Say, Briggs, why do you keep emphasizing the one hour?”
Glad you asked, friend. Because we go back to our millennia of evidence where people were not just exposed to one hour, but to lifetimes of candle dust. And there is no evidence in favor of the idea we are smarter than our ancestors; indeed, all indications are we are worse.
Maybe we should burn more candles.
To support this site and its wholly independent host using credit card or PayPal (in any amount) click here