But this time criticism of the latest we-are-dooomed scenario has extended beyond the usual cluster of sites. Even Der Spiegel picked up on it: Globale Erwärmung: Forscherkrieg um Klimastudie.
The relevant passage, which I used Google Translate to provide, is this:
More serious the allegation several critics, the data base of the study was chosen so that a certain result come out. Hsiang and his colleagues had “comparing apples with rollerblades”, wrote about William Briggs, statistics professor at the U.S. elite Cornell University, dripping with sarcasm in a blog post . “Data of past Tuesday will be just as meaningful as that of 8000 BC.” At the end are “many great graphics” and a result came out: “Hot, dry weather is bad for us.” Therefore, mocks Briggs, also draw everyone into the hot from the cold Michigan South Carolina – “to be where the action is.” “Complete nonsense,” was still the kindest description of the study.
No, no. My kindest description was to say that Hsiang and his co-authors put in a staggering amount of work. No faulting their diligence, he said sincerely.
But hard work doesn’t always imply good work (as regular readers of this blog will attest!).
Here is the succinct summary of Hsiang: they mixed physical measures of the atmosphere and land from widely disparate times and places without accounting for the uncertainty inherent in each or in their combination; they defined “violence” haphazardly, intent to let it be rape at one location and “leader removal” at another. They cobbled the whole together into a—let me charitable—creaky statistical model which was able to produce a wee p-value, defined as a p-value less than the magic number (0.05).
Additionally, every site and data source they used did not show increasing “violence” with increasing inclement weather (people do not experience a climate, but moment-to-moment weather). It was only after mixing the stew of the sources and kinds of “violence”, all said to be just plain “violence”, into their model were they able to “prove” what they hoped would be true.
Suppose they were interested in rape. What they should have done is to look at rape statistics from localities all across the globe, accompanied by those meteorological variables which they could show, or plausibly show, are directly responsible for rape. They’d have to answer questions like, Why isn’t rainier weather correlated with less rape? After all, rapists like to keep dry, too.
But this alone would not have been good enough to show a connection. After all, rape rates are measured with substantial error which depends on location. These uncertainties would have to be “carried forward” in their models. And then culture plays an enormous role in rape: these variables would have to be controlled for.
Of course, the weather/climate variables are also subject to large measurement uncertainties, which also depend on location and time. Time itself is highly problematic. Cultures change through time, immigration and emigration have to be understood, technology changes things (e.g. cameras may dissuade rape). And on and on.
And these are just the bare minimum to be able to say anything about just rape. What about “leader removal”? Same set of considerations, only now expanded for politics, that simple subject. And so forth for the other kinds of “violence.”
Then, finally, when all this is done, they’d have to look at the “opposites” of these variables. It could be that “leader removals” increase with temperature, but it might also be that people are happier, politically speaking. It was not joke about people moving from the North to the South, where it is much hotter: it’s happening with increasing frequency and people seemed pleased about it. They do not seem to be turning to violence at their new addresses.
Just because it is easy to dump a bunch of numbers into a computer does not mean that it should be done. The software is forced to give you an answer, but that does not necessarily mean the answer is what you hope it is. That this paper showed up in Science says lots about how easily influenced even our best and brightest (but see Belloc’s exception) can be by fad and bias.
Update More from the Der Spiegel article (at the end):
Storch also makes allegations of the “Science” Editorial: “The peer-review process did not work you should have critical evaluators, which was obviously not the case here..” A spokeswoman of the journal, however, can see nothing but good in the debate. “Science is a self-correcting process,” she wrote in an e-mail. Researchers published their results so that they confirmed could be refuted or corrected. “In this way science makes progress.”
You bet it is, baby. Consider this a massive self-correction.
Is anything so bad that it can’t be made worse by Germans? OK, too broad: is there anything so bad that it can’t be made worse by northern Germans in traditional Prussian-influenced areas? Didn’t think so.
Good to see critics get any print at all. And I appreciated your expounding some on the basic mechanical problems with data and controls – that’s something even us non-statisticians can appreciate. The thing for us amateurs to remember: just because you can run some numbers through the statistical meat grinder doesn’t mean it makes any sense to do so.
Finally, this ‘self-correcting’ idea seems, I dunno, kind of mystical. Are we to believe in a sort of Dead Hand of Science, keeping science on track regardless of the wishes of the practitioners? Or, rather, is the mechanism more truly a result of nurturing an adversarial culture, where *disproving* claims is an honorable and, indeed, indispensable duty?
Kind of like the press and politicians?
The phenomenon in which bright otherwise erudite people produce invalid conclusions from research projects can be traced to a gap in their training that is produced through the ignorance of their professors of recent advances in logic. A piece of evidence for this contention is the course notes for Logic I at MIT in the year 2005. In these notes, the professor tells his students that logical principles by which one can go from specific instances to general rules (aka scientific theories) are unknown. These principles, the so-called “principles of reasoning” are precisely what are required in order for valid conclusions to be drawn from a research project. A professor at one of the most elite of research universities was advising his students that logically valid conclusions could not be drawn from a research project and this advice was wrong!
I suppose dangerous highway driving is auto-corrected (snort!) by accidents too. How long are journalists (in the broad sense of popular magazine writers and Science editors) going to use the scientific process excuse when foolishness could be avoided?
Some Mirrors are better than others.
This is definite proof that one should eschew Google translate for such sentences as Deshalb, spottet Briggs, ziehe auch jeder vom kalten Michigan ins heiße South Carolina – “um dort zu sein, wo die Action ist”.
Google responds: Don’t be Evil.
Or we’ll sue you for patent infringement.
May I point out this excellent translation by my colleagues from our international channel? http://www.spiegel.de/international/world/scientists-debate-the-relationship-between-climate-change-and-violence-a-915780.html
This should be much better than the Google transmogrifier.
Greetings from northern Prussia!
Pingback: Facts, Theory and What is Important? « Yard Sale of the Mind