I had the knife at my throat after reading a paper by Preti, Lentini, and Maugeri in the Journal of Affective Disorders (2007 (102), pp 19-25; thanks to Marc Morano for the link to World Climate Report where this work was originally reported). The study had me so depressed that I seriously thought of ending it all.
Before I tell you what the title of their paper is, take a look at these two pictures:
The first is the yearly mean temperature from 1974 to 2003 in Italy: perhaps a slight decrease to 1980-ish, increasing after that. The second pictures are the suicide rates for men (top) and women (bottom) over the same time period. Ignore the solid line on the suicide plots for a moment and answer this question: what do these two sets of numbers, temperature and suicide, have to do with one another?
If you answered “nothing,” then you are not qualified to be a peer-reviewed researcher in the all-important field of global warming risk research. By failing to see any correlation, you have proven yourself unimaginative and politically naive.
Crack researchers Preti and his pals, on the other hand, were able to look at this same data and proclaim nothing less than “Global warming possibly liked to an enhanced risk of suicide.” (Thanks to BufordP at FreeRepublic for the link to the on-line version of the paper.)
How did they do it, you ask? How, when the data look absolutely unrelated, were they able to show a concatenation? Simple: by cheating. I’m going to tell you how they did it later, but how—and why—they got away with it is another matter. It is the fact that they didn’t get caught which fills me with despair and gives rise to my suicidal thoughts.
Why were they allowed to publish? People—and journal editors are in that class—are evidently so hungry for a fright, so eager to learn that their worst fears of global warming are being realized, that they will accept nearly any evidence which corroborates this desire, even if this evidence is transparently ridiculous, as it is here. Every generation has its fads and fallacies, and the evil supposed to be caused by global warming is our fixation.
Below, is how they cheated. The subject is somewhat technical, so don’t bother unless you want particulars. I will go into some detail because it is important to understand just how bad something can be but still pass for “peer-reviewed scientific research.” Let me say first that if one of my students tried handing in a paper like Preti et alia’s, I’d gently ask, “Weren’t you listening to anything I said the entire semester!”
Any method that attempts to correlate the two raw data series will end in grief, for the obvious reason that the pictures look nothing like one another. Temperature more or less increases and the suicide rates peak—at different points for males and females—then trail off. The authors never bother to hazard a guess why this is so. But I can ask, is it a coincidence that the highest male suicide rate coincided with Italy’s loss to Brazil in 1994?
Anyway, because the raw series are obviously unrelated, and the authors need to have a paper showing a global warming connection, they have to manipulate the data in some way. The method our authors applied was this: They first fit a trend model to the temperature series (a line), and a quadratic polynomial to the suicide series. Why quadratic? Just looking at the curved, solid line on the plots shows that the quadratic fits terribly, especially for women. But, like the authors, let’s ignore that.
The fact is that no model is needed for either series because these data are the actual data. There is nothing hidden here, nothing that remains uncertain that we need to model with a probability distribution. It’s true that we can model the relationship between the series, but that’s not what they did. They modeled each series by its lonesome, and they modeled it badly.
This is an important point, but a minor one, because their goal was not to actually model the series after all, but to produce the “residuals” from such models. Residuals are the numerical difference between the actual series and its model. The authors then tried to model the residuals from each series to see if these were related. No dice. So they had to manipulate the data further to produce the result they wanted.
What trick did they next try? A “Gaussian low-pass filtering procedure” mainly used in eliminating noise from images (they never show nor explain their algorithm nor attempt to justify its use). You might think of this as a way to smooth the residuals. I have elsewhere on this blog showed how smoothing (creating running means is a smoothing method) can induce correlation between two smoothed series even when those series are absolutely unrelated. This means our authors are more likely to see an effect that is not there. Did they?
No, blast it! Even after all this, they still could not find a publishable p-value connection between the two twice-modified series. But they still had to—had to—prove their theory correct. So what did they do? Anybody?
Yes, they manipulated the series a third time! The trick they used was the oldest in the book: they threw out the data that did not fit their preconceptions! Think I’m joking? Here it is in their own words (recall that all the other manipulations have already taken place):
[W]e were able to identify deviant points (or statistical outliers) in our series; by excluding deviant points (no more than 1-2 per series), a sensitive increment in the association between our measure of temperature and suicide (Table 1). The exclusion of deviant points is reasonable if we consider that we do not expect temperature anomalies and suicide residuals to be coupled fully, unlike a univocal cause-effect relationship: it is self-evident that a human variable as suicide occurrence is linked to several causes that include also a simple stochastic occurrence, which may deviate from the general trend due to an underlying main variable. The detection of a deviant point is therefore a useful exercise to take these possible stochastic non-causal occurrences into account in a critical way. The exclusion of deviate points made even more evident that the links between anomalies of temperature and suicides concern the warmest months…
As pure a stream of unmitigated bullshit I have rarely seen short of a communist party broadside. There is nothing in this paragraph that makes the slightest sense; each time I read it I am exasperated. Note carefully how they try to avoid admitting the fix by using the words “our measure of temperature and suicide” which here literally means “the two series we created by several different methods but which are not temperature and suicide.”
The whole point of this gibberish is to say, “We could not find a signal before we manipulated our data, nor could we find it after we manipulated it twice successively. But were were finally able to produce what we were looking for by throwing out the points that did not fit our needs.”
Only after they tossed the data (and after the manipulations) were they able to get publishable p-values. And not too many either: they only found a relationship in a couple of months and really only in males. As cheaters they stink. But not entirely. They did (12 + 4)*2 = 32 separate tests (one for each month and season once per sex), and they should have adjusted their p-values up to account for the multiple chances for success. They did not. Well, no time be ethical now. If they did adjust, their findings would vanish.
The only question I have is: did they cheat knowingly or were they so anxious to justify their fears of global warming that they actually believed that anything they could do to the data would be a service to humanity? Sigh. Probably the later.
The comedy really begins when the discuss their findings. Our authors have learned to write badly as all academics must if they want to see their names in print. Try this fragment
At a first glance, more suicides in the months with a greater thermometric discomfort can be attributed, at least partially, to the effect of global warming on survival after attempt. Very simply, extreme temperature might increase the negative impact of body self-harm on the chance of surviving after impact…
“Thermometric discomfort”? “Negative impact of body self-harm”? Oh, Lord. At least they later admit that “No doubt that thermometric discomfort can cause stress even when associated with increasingly cold temperatures.” By which they mean, cold weather can be awful too. Fewer suicides might—might—be found in winter because people are stuck inside and “are more likely to get an enhanced surveillance by their families.”
The authors are willing to grasp at any reason, any connection no matter how tenuous, that shows they are right. Thus,
As correctly pointed out by an anonymous reviewer, the same mechanisms affecting climate (anthropogenically-induced multi-pollution) are also affecting human health, as seen in the increased incidence of cancer in younger people and in the earlier onset of neurological disease in adults in the Western world…This may, as well, increase the risk of suicides, which is itself enhanced in these illnesses…leading to a spurious correlation with anomalies of temperature that are interrelated with all the other effects of multi-pollution.
“Multi-pollution”? Yeesh. Stick with me, it gets worse…
Moreover, the waste of natural landscapes and the deterioration in flora and fauna related to multi-pollutions and its effects on the natural world might further reduce the sense of satisfaction had happiness in people, excursion in natural environments representing an opportunity to buffer the negative stress of urban life.
What’s the word that describes wanting to cry and laugh simultaneously? That word is me all over after reading that.
Each sentence in this work is a gold mine of unintended comedy and shoddy work. I could write a book on How Not To Do Research based on just this paper. However, I am growing weary and have probably tested your patience far longer than I should have. So we will leave off with their overall conclusion:
An improvement in the ability of communities to adjust to temperature changes by implementing public health interventions may play an important part in preserving the wellness of the general population, and also in limiting the worst consequences of suicidal behavior.
” Worst consequences of suicidal behavior”? Oh, I give up.
You may want to fix these:
“How did they do [sic], you ask?”
“How, when the data look absolutely unrelated, where [sic] they able to show a concatenation?”
“This is an important point, but a minor one, because there [sic] goal was not to actually model the series after all, but to produce the “residuals†from such models.”
Gregorio,
Good grief! I am nearly as bad as the paper I picked on! I could use the excuse that my indignation blinded by editing, but it’s more likely I am just a poor writer.
Thanks very much.
There’s an episode like this in “Asterix and the Romans”. The Roman general has forced Getafix the druid to make magic potion for the Romans. He’s first in line and after taking his potion he rushes out to perform some superhuman feat. He seizes a monstrous boulder. No joy. “Maybe being overambitious”. A slightly smaller boulder. Still no joy. “Perhaps it’s not working fully yet”. A smaller boulder. Grunts, sweats. Nothing. Eventually he grabs a rock about the size of his fist, raises it over his head and shouts, “I have superhuman strength!”. And all the while his beard is growing…
It’s one of my favourites.
Rich
While there are humorous aspects of this piece, it is really deadly serious because we are witnessing this kind of nonsense throughout the climate science arena and its peripherals. Is this really so much different than the dishonest manipulations pulled by the advocates to preserve the iconic status of the hockey stick? Not at all.
Willis Eschenbach recently argued, persuasively I think, in Roger Pielke’s Prometheus blog that the problem is not in the few who try to game the system. There are always those that try. Rather the problem is in the broader sceintific community that permits the erosion of standards as long as the “scientific” result fits the politics. Want another example? Take a look at the recent US Climate Change Science Program “Unified Synthesis Project.” Pure advocacy masquerading as science, and funded by taxpayers as well. These obvious examples are only the tip of the iceberg. The corruption of the scientific process continues on a much larger but more subtle, scale as confirmation bias drives a vast research effort.
Will science recover from this assault on the process carefully put together over centuries? I don’t really know. Winners write history.
It should be surprising that such a bad paper, which is not really flattering of AGW catastrophism is publish.
Sadly it shows the corruption of the peer-review process. With the AGW phenomenon, it becomes ever more evident that any paper linking catastrophic and AGW is accepted whatever the quality of the paper really is. In other word if it is catastrophic it is publish no matter what is wrong with it.
With the number of paper being submitted for approval, it is not hard to see that reviewer which are occupied with their own work, go over their review very quickly and that don’t look further if the conclusion agree with them.
Glad to see you recommending eyeballing as a first step. I’m sick of certain statisticians (pros and wannabees alike) saying that your eyes can lie and you need to put the data through a filter to see what’s really there. Amazing how easy it is to fool yourself and justify it with pure BS.
Personally I like hot weather. It’s cold, wet and dark that depresses me. And I somehow suspect I’m in the majority there. And funnily enough, the flora and fauna in my garden seems to like sunny weather too.
Agree with you, JamesG.
If you can’t see a correlation it probably is NOT there.
If you can only see it after putting it through a statistical filter (especially one that you don’t understand) then it almost certainly is NOT there. True of lots of AGW stuff.
The wonderfully named, SAD (Seasonal Affective Disorder), is based on low light and cold being a trigger to SAD-ness.
Surely this is a spoof article to demonstrate that the editiors of the Journal are gullible idiots? One test would be to see if their earllier papers were equally ridiculous.
They’re nuts. Must be global warming.
It is possible that your eye can fool you, but it doesn’t happen often, and only for subtlest of effects. There is nothing—not one thing—subtle about this data. They are so different that it is ludicrous to suggest a connection.
And it’s not just a filter that they used, but the filter on the residuals of their crappy regression model (the quadratic “fit” on the suicides). Take a close look at the fit and you’ll see how the magic happens.
Define the residual as Actual data – Modeled data. You’ll notice that the residuals are negative numbers (for males or females) at the start of the series and then become positive numbers near the peak suicide rates; after this the residuals hover near 0.
Did you get it? This is the big trick. What do you call a series of numbers that start out small (negative here) and then grow to something large? We call it an increasing series, or something like a line.
Temperature is also an increasing line. Suddenly, because of the nonsensical modeling, we have two series, temperature and residual suicides, that are increasing together! Then comes the filter to make the lines straighter, and thus the spurious connection stronger. Tossing out unwilling data is just icing on the rotten cake.
That’s a bad metaphor, I know, but I am sputtering. I am simply aghast at this paper. Who in God’s name reviewed it? The awfulness of their method is so obvious that even a Harvard grad can see it.
Dr. Briggs, thank you for the analysis. Please don’t let this kind of nonsense get you down. We laymen of the world desperately need minds like yours to counteract this nonsense.
Dr. Briggs,
Thank you for this post. I love to see bad science (and bad reporting) held up to ridicule.
I also love the tag line for the blog – “All manner of statistical analyses cheerfully undertaken.” I am hoping you are willing to undertake an analysis for me.
Nature published a story about the NAS report titled “Academy affirms the hockey stick.” http://www.nature.com/nature/journal/v441/n7097/full/4411032a.html
However, it is also my understanding Gerald North said the NAS panel basically agreed with the Wegman Report. I do not know of anyone who thinks the Wegman Report affirms the hockey stick. http://www.climateaudit.org/?p=2322
I commented on this on Nature’s blog: http://blogs.nature.com/climatefeedback/2008/08/more_for_the_annals_of_climate_1.html#comment-94817
Here’s the question I want you to analyze:
Did the story in Nature accurately reflect the findings of the NAS panel? On what percentage of controversial questions does North agree with Wegman?
I would love to see a full blog posting on this subject. I think many others will find it interesting as well.
Of thread to some extent, but to Ron Cram, did you see the Bishop Hill’s effort at explaining the last several years involving Steve McIntyre and Wahl & Ammann. You can find it on Bishop Hill’s blog or ClimateAudit, if you haven’t seen it already.
Ron,
I don’t think anybody could top the work that Steve McIntyre has gone on the infamous hockey stick. Wegman’s review is also pretty good.
Nobody fools themselves as easy as a scientist.
There is the famous example of a nonsense paper full of gibberish being submitted and accepted by a post-modernism journal. This example is used to discredit postmodernism.
However this paper was submitted to and accepted by a scientific journal. The hockey stick paper with its nonsense statistics was submitted to accepted by Nature.
Is there something similar about these separate cases?
Dr. Briggs, you refer to the authors as “crack researchers.” But is it possible they are really “crack ho’s?” That would explain the poor statistics, and the tortured logic they utilise.
If the peer reviewers were truly peers–other crack ho’s–that would explain the current state of climate papers being published today.
Even after all that sophistry nothing is proved. As we are regularly, and correctly, told, correlation does not mean causation and your aside about Italy losing to Brazil is more likely to account for some of the suicides.
Mr Simpleton,
Thank you for laughing at my joke. I was beginning to worry it was forever lost.
William,
Of course I agree with you regarding Steve McIntyre’s work on the Hockey Stick. That really isn’t the question.
The question is: Will you analyze the reporting done by Nature regarding the NAS report? If I understand the situation rightly, you were part of the proceedings and you would have some background (and right) to express yourself on the issue of whether Nature reported the story correctly or not.
I am not nearly as good at this kind of thing as you are but here are some questions I have:
* How many disputed issues did the NAS panel assess?
* On the disputed issues, how many times did they agree with MBH? How many times did they agree with M&M?
* What percentage of agreement would be necessary for the title “Academy affirms Hockey Stick” to be true?
* Did the report by Nature accurately reflect the findings of the NAS panel?
* If Nature’s report does reflect accurately the NAS report, how does that fit with Gerald North’s statement that the NAS panel basically agreed with Wegman?
McIntyre could do this kind of thing, and has. The problem is that many people will not give his analysis any credence only because it appears McIntyre has a vested interest in the outcome of the analysis. But if the analysis is done by someone who worked on the NAS report, it carries much more weight.
Will you please reconsider?
Tony,
Yes, I have read Bishop Hill’s post and commented on his blog. I also commented on Alex Lockwood’s blog about Bishop Hill’s blog.
The issue here is that I have commented on Nature’s blog that their story titled “Academy affirms the Hockey Stick” was misleading and “went into the realm of active disinformation.” I asked them to correct their story. See http://blogs.nature.com/climatefeedback/2008/08/more_for_the_annals_of_climate_1.html#comment-94817
Unfortunately, for the time being, Nature is standing by their story. But it is my hope that with enough people talking about the facts of the case that Nature will set the record straight.
As it stands right now, Nature has lost its credibility.
Ron,
Nature has long ago tainted their credibility, so this is not surprising.
Now, Mann’s Hockey Stick analysis is silly, and Wegman took him to task for it. The other committee agreed that the analysis was silly, but that the series itself might still be true, or that Mann’s result was consistent with the theory. (This is a weak, rough summary, I know.)
Well, here’s the big shock: Any observation you make is “consistent with” the AGW theory. This is because that theory does not say any particular observation is impossible.
This logical possibility is what allows Nature and other journals to insert their editorial opinion into what otherwise would be a factual article. To say that Mann’s finding still “might” be true or is “consistent with” AGW are true statements. They are just incredibly improbable.
I can see that I’ve done a bad job explaining what I mean. I’ll try and clean it up in another post because the point is actually a general one and applies to any theory.
Briggs,
I agree up to a point. Your “rough summary” is roughly accurate – considerably more accurate than Nature’s reporting.
I do not see how the NAS panel can say Mann’s construction “might still be true” and that Nature can report it as the NAS panel says Mann’s construction is true. That is just not honest reporting.
I look forward to your next post on the subject.
Great article.
I was once asked to do a correlation analysis of some very random looking data. I told them you usually know the conclusion before you start. If the correlation looks real bad, it mostly is.
I’ll be back.
During happy days before the outbreak of World War II, some graduate students and junior faculty at Princeton U collaborated to write spoof academic manuscripts to see if these could pass peer review. Several were duly published.
One contributor was post-doc Ralph Boas, who would become a prof at Northwestern and had a turn as head of the AMS; another was Lyman Spitzer, who would become an eminent astro-physicist and pioneer of nuclear energy; a third H. P. Robertson, junior math faculty and cryptanalyst; and a fourth kindred spirit was scientific generalist John W. Tukey.
And so it came to pass that a leading math journal actually published a short paper that arrived at the curious conclusion, 0=1.
Beau,
That’s fantastic. I love that story. Is there anywhere to read more about it?
Sorry I didn’t understand most of what is above. Is there a link between global warming and suicide rates or not?
Oh sure, Provocateur. People have been dropping like flies. Be especially careful walking by tall buildings.
Thanks, I need to login to sciencedirect to get the paper. Trying, sounds good. I have to produce a yr8 paper on how global warming is detrimetally effecting the people of the world. If this is no good I’ll have to go back to the mindless gibberish the IPCC produce like everyone else.
Thanks Again
Provocateur…
“A contribution to the mathematics of Big Game Hunting” was included in the Selected Papers of Lyman Spitzer (1995) Princeton Univ. Press. Spitzer asked Tukey if he had objection to its inclusion. The putative author was H (as in Henri) Petard. It was published in American Mathematical Monthly, 45, 446-447. (Henri married Betti Boubacki and formal wedding invitations were printed up.)
I do not have the cite readily available on the 0=1 paper, clear enough for me to appreciate the humor. It was probably published in the same journal, likely under the name E. S. Pondizcery. (Erastus Stanislaus?) His affiliation was the Royal Institute of Poldavia. His acronym was thus ESP-RIP.
The authors were, I am told, inspired by surprising papers on extra sensory perception by Duke University E. B. Rhine. Rhine reported experiments establishing ESP, but this owed to his exclusion of some of the results, when someone was not feeling full of ESP. Rhine’s work was thus buttressed on biased data analysis.
Frank Smithies, a post-doc at Princeton during 1937-38, was a member of the circle during its the first year, before returning to Cambridge.
Tukey and Boas persuaded a Congressman from Wyoming to enter into the Congressional Record a spoof mathematical discovery attributed to one of his constituents, an engineer working on a WPA dam.
The timeframe of this fun was likely 1938-42. It might have taught lessons, namely much can make it through peer-review.
James B. Fisk was a serious physicist, in time head of Bell Labs and headed the the US delegation in nuclear weapons negotiations circa 1958-59. In the late 1930s, Fisk owned stationary that read something like: Society for the Advancement of Useless Research. (This is from memory and may not be precise.) Fisk sent a note to Robertson and Tukey on this stationary, co-signed by physicist J. R. Pierce, saluting the big game hunting article. So the big game article must have been considered good-fun, in its day.
Beau,
That’s fantastic, thanks very much.
The sense of humor index has dropped in recent years. I don’t recall many chuckles from sociologists etc. after Alan Sokal published his fake paper, for example.
If you tried it now, even in math or physics, you run the risk of being condemned.
Dear Dr. Briggs: One reason I find it easy to accept you are a statistician is the phrase “all analyses cheerfully undertaken.”
And it makes sense you could be drawn to a popular, important topic of current times, such as the theory of catastrophic human induced global warming owing to CO2 or other gases.
The environment seems a broad field in which many strong inferential claims seem to get made, based on modest or narrow or uncertain data. Politics and moral opinions get added into a combustible mix, in a way that would not readily happen with so many other fields of human inquiry. (It would seem hard to take some mind-numbingly arcane area of debate within many academic fields and emerge with an earth-shattering fear, such as the ozone layer being eaten up or a few industrial chemicals are causing male abundance of sperm to diminish, a fear championed by the former VP in the mid 1990s, before he cheerfully embraced AGW.
Eco-scientists would like to believe they wear white hats, since busy beavering away, saving the earth. And because they are so laudable, perhaps this could explain why there are woefully low standards within this subject area for responsible, careful analysis of data.
Skepticism is essential to good science, but not very naturally popular within environmental science, given its moralistic contexts and political customers. Politicians want to grab something published by an environmental scientist and run with it; narrow interest groups are happy to similarly gain and profit. Vain environmental scientists are happy to have their claims receive spectacular endorsement. There is potential for epic farce.
This is why environmental science can use more statisticians (philosophers of science) examining it’s theories and claims. Tukey was active in environmental issues for several decades. Bjorn Lomborg must have been attracted. Junk Scientist Stephen Milloy has a masters in biostatistics from Johns Hopkins and seems to have a sense of humor; he seems to be a comedian of environmental data, a modern-day William Hogarth or H L Mencken. Since environmental studies are an active arena for politicians, they arguably warrant political satire.
Given this overall context, it seems plausible that Dr. Briggs could cheerfully enter the field of global warming and might be able to find beliefs and assumptions that could reasonably surprise him, after careful analysis of their foundations. The environment is a topic deserving of scrutiny by statisticians, for the sake of the common good.
Beau, Hear hear!
Beau,
All your points are excellent. We welcome all suggestions for analysis here, so feel free to chime in anytime.
Ray Bradbury wrote about a similar correlation between heat waves and murder many years ago way before AGW. Some insurance analysts try their luck at saving the world. Sir Hitchcock did a 30 minute TV show about it in 1956 called “Shopping for Death.” I don’t consider it the best Hitchcock has done, but a fun view from the past:
http://www.hulu.com/watch/780/alfred-hitchcock-presents-shopping-for-death#x-0,vepisode,1