William M. Briggs

Statistician to the Stars!

Category: Links (page 38 of 78)

The statistics of both climatology and meteorology.

What’s Wrong With Hypothesis Testing: Reader Question

I received this email from long-time reader Ivin Rhyne, which is so well put that I thought we should all see it (I’m quoting with permission):


I just got back from a conference on historical economics and was absolutely bowled over by the repeated usage of t-tests and p-values as the arbiter of whether an hypothesis is false or not. Allowing for the subtleties of “reject” vs. “unable to reject” my question is more numerical.

My personal understanding of regression analysis to test “fit” a model to the data is as follows:

1. Form a hypothesis
2. Gather some data to test your hypothesis
3. Translate your hypothesis into a form that is mathematically testable (let’s assume OLS regression is a good mathematical expression of your original hypothesis)
4. Using part of your data, calibrate the OLS by running it to get some numerical parameters that then become an intrinsic part of your hypothesis
5. Using the rest of your data (the part you DIDN’T use to calibrate the model) you actually insert the data points for the independent variables and then compare how closely the dependent variable matched the actual values.

ALL of the papers presented at the conference stopped at step 4. Their test of the hypothesis was simply whether the model could “calibrate” to the data in a way that generated coefficients that had “acceptable” p-values.

My questions are as follows:

1. Have I missed something and in fact theirs is the correct approach to hypothesis testing of social science data?

2. If I am right (in principle) about how to test hypotheses, can you point me toward (or perhaps even better lay out in your blog) what kind of test is appropriate for step 5 described above for an OLS regression?

As always, I appreciate your insights.


This nails it. I have rarely seen a sociological or other “soft science” paper venture beyond Step 4. A few make a stab at Step 5, but usually in such a way as to dissolve the force of this Step.

It’s cheating, really, and done by formulating several models, usually the same underlying OLS but with different sets of “regressors” for each model, and then each is tested (crudely) via a Step 5. The one that’s best, or the one that is best within the subset matching an author’s desires, is the one that makes its way to print.

I hope you can see that doing this is just the same as skipping Step 5. Or it’s equivalent with Steps 1 – 4, but with a “meta” model. Regardless, it is using the data you have in hand to massage a model into a shape that is lovelier to the eye. It is not an independent test of your model’s goodness.

Ideally, there is no Step 5—all data you have in hand should be used to construct your model—but there should be a Step 6, which is “Wait until new data comes in and test the model predictively.” All physical sciences do Step 6—with the exception, perhaps, of climatology, where it’s the “seriousness of the charges” that counts.

Passing a Step 6 does not, of course, guarantee the truth of a model. Just look at the Ptolemaic systems of cycles, epicycles, semi-cycles, and so on ad infinitum. Wrong as can be, but still useful. The model passed Step 6 for centuries, which is one of the reasons few thought to question its truth. Don’t mess with what works!

From this history we learn that passing Step 6 is a necessary but not sufficient condition in ascertaining a model’s truthfulness. Spitting out a p-value (Steps 1- 4) that is less than the magic number is not even a necessary condition; and anyway, the p-value was purposely designed not to say anything about a model’s truthfulness.

We must remember that, for any set of evidence (data), any number of models can be made to explain that data; that is, you can always find models which fit that data. Simply touting fit—as in Steps 1 – 4, and the p-value’s main job—is thus very weak evidence for a model’s truth.

Why aren’t more “Step 6″s being done in statistics? It’s not that it’s difficult computationally, but it is expensive and time consuming. It’s expensive because it costs money to collect data. And it’s time consuming because you have to wait, wait, wait for that new data. And while you’re waiting, your wasting opportunities for “proving” new theories.

Much more to this, of course. For example, why do some models work even when people flub the steps? Because models are chosen with reference to external probative information. We’re obviously just at the beginning of a discussion.

What Is A “Statistically Significant Trend”?

Longtime reader Nate Winchester found a discussion—among the many, many—of global warming data revolving around statistics, from which we take the following snippet:

You just do the statistics on the data. If you calculate trends over a short period you don’t get statistically significant trends, and over a longer period you do get statistically significant trends. This is true for almost any real life data, and how long it takes for trend to show up over short term non-trended variation will depend on the data.

In the case of global temperature anomalies, it turns out that the trends in temperature become statistically significant over scales of roughly 15 to 20 years or more, and lack significant trend over shorter scales. That’s just a description of what global anomalies are doing.

Where this quotation originated is not important; probably you can find one nearly identically worded at any site in which the subject of climate change arises. But it is a useful comment, because it betrays a standard misinterpretation of statistics which we can here put right.

Suppose in front of you is a picture of a number of dots, one per year arranged sequentially, each dot representing, say, a temperature. Obviously—yes, truly, obviously—those temperatures, assuming they were measured without error, came from somewhere. That is, something, some physical process or processes, caused the temperatures to take the values they did.

They did not appear “randomly”, if by use of that word you mean some vague and mysterious metaphysical engine (run by quantum gremlins?) which spit the temperatures out for humanity to discover. But if by that word you merely mean that you do not know or do not understand what physical process caused the temperatures, then you speak intelligently.

Our second supposition requires us to weakly anthropomorphize either all, or individual portions, of the dots. You have to squint at the collection and say to yourself, “Say, if I draw a straight line running amidst the dots between year A and year B, most of those dots will lie close to the line, though only very few will touch the line.” You are allowed to draw various lines through the dots, some pointing upwards, some downwards, as long as all the lines connect head to foot, starting at the first year and ending at the last.

Once done, you can reach into your bag of statistical tricks and then ask whether the lines you have drawn are “statistically significant.” The first step in this journey to amazement requires you return to the word “random” and invoke it to describe the behavior of the dots not lying on the line. You have to say to yourself, “I know that nature chose to make the temperatures lie on this line. But since they do not lie on the line, only close to it, something else must have made the dots deviate from the line. What this cause is can only be the normal distribution.”

In other words, you have to say you already know that nature operates in straight lines, but that something ineffable steers your data away from purity. The ineffability is supplied by this odd who-knows-what called the normal distribution, the exact nature and of motivations of which are never clear.

Another thing that isn’t quite clear is the slope of the line you drew. It is a line, though; in that you are certain sure. But perhaps the line points not so nearly high; rather, it might lie flat. Must be a line, though. Has to be. After all, what else could it be?

Now, with all these suppositions, surmises, and say-whats in hand, you feed the dots into your favorite statistical software. It will churn the dots and compute a statistic, and then tell you—the whole point of the article has now come upon us, so pay attention—it will tell you the probability of seeing a statistic larger than the one you actually got given your line theory and your ideas about randomness are faultless (I ignore mentioning infinite repetitions of data collection).

If this probability is small, then you are allowed to say your line is “statistically significant.” Further, you are allowed to inform the media of this fact, a tidbit for which they will be grateful.

Of course, saying your imaginary line(s) are “statistically significant” says nothing—not one thing—about whether your line(s) are concrete, whether, that is, they describe nature as she truly is, or whether they are merely figments of your fervid imagination.

The best part of this exercise, is that you can ignore the dots (reality) entirely.

Global Warming Causes Death

Work is catching up to me this week, so today only the briefest of reports, with the promise of a return to regularity after this weekend. Also keep those post suggestions coming everybody!

Thanks to the Daily Mail, we learn of new “research” which says that global warming causes cancer.

Melting glaciers and ice sheets are releasing cancer-causing pollutants into the air and oceans, scientists say.

The long-lasting chemicals get into the food chain and build up in people’s bodies – triggering tumours, heart disease and infertility.

It had to happen. Global warming has been proved, in the same manner as this new paper proves its dread effect, to cause every other horrific kind of premature death, from swarming insect attacks, to rampant prostitution and its natural aftermaths, to unstoppable epidemics of this bacteria and that virus.

But nobody thought to put the Big C on the list until Donald Cooper of the United Nations Environment Programme. His mental efforts deserve some kind of prize or award, surely. It’s true Cooper’s invention is not the most imaginative cause of death caused by climate change. That distinction, even though I say it myself, goes to me and my number two son, after we proved conclusively that zombie attacks must increase because of global warming.

It’s also so that it was only a matter of time that cancer showed up as an aftereffect of a half degree Centigrade rise in global average temperatures. Cooper real feat was being first, for having the guts to cross the line and say what others wanted to, but did not. So this one’s for you, Coop. You have provided the last legitimate fright.

That also means there is nothing else; at least nothing else conventional and semi-realistic (zombies don’t fall into either classification). Even increased waves of madness have been predicted to arise because of global warming, the waves composed of human bodies crashing to the ground in suicide attempts (yes, suicides are supposed to increase because of global warming).

Researchers can only gain notoriety by predicting the new, bizarre, or shocking. All the regular routes have been taken. Therefore, the question we have before us today is: what new deadly realistic thing will global warming be said to cause to increase? Let’s get the predictions noted down here, all in one place, so they are easy to verify later.

The only rules are: you can’t use something already in print, and you should try hard to tie your dread cause of death to a funding source, so that when you use the phrase “more research is needed”, you see the money flow.

NASA’s New Life, Global Warming Survey

Quick post today: busy with grading.

NASA’s Aliens

You will have heard of NASA’s press release, which all are characterizing as “redefining life as we know it.” As I read about this on other sites, it appeared that NASA’s astrobiology group had discovered, on Earth, a bacteria that natively replaced arsenic with phosphorus in its DNA. But NASA’s press release reads:

The newly discovered microbe, strain GFAJ-1, is a member of a common group of bacteria, the Gammaproteobacteria. In the laboratory, the researchers successfully grew microbes from the lake on a diet that was very lean on phosphorus, but included generous helpings of arsenic. When researchers removed the phosphorus and replaced it with arsenic the microbes continued to grow. Subsequent analyses indicated that the arsenic was being used to produce the building blocks of new GFAJ-1 cells.

The key issue the researchers investigated was when the microbe was grown on arsenic did the arsenic actually became incorporated into the organisms’ vital biochemical machinery, such as DNA, proteins and the cell membranes. A variety of sophisticated laboratory techniques was used to determine where the arsenic was incorporated.

This makes it look like the bacteria, in some sense artificially, replaced its phosphorus after being deprived of that element, and that actual in situ versions of the arsenic-only microbe were not found. And it also appears that only some, but not all phosphorous was replaced.

Remarkable no matter which way you look at it; fascinating. Any biochemists or biologists out there who can explain this more in depth?

Global Warming Survey

I received this press release, in which some of you might have an interest. My memory tells me that I was one of the people who filled out the survey, but it was a while ago and I can’t recall.

The Greenhouse Gas Management Institute and Sequence Staffing are pleased to announce the results of The 2010 Greenhouse Gas & Climate Change Workforce Needs Assessment Survey, our second annual international survey to determine the latest workforce needs of the greenhouse gas and global climate change industry.

The full results are here. Very slick presentation; not much substance to my eye.

1. Climate change remains an emerging field where practitioners rise quickly through the ranks.
2. GHG training gets high marks overall, but serious reservations are noted.
3. U.S. facilities are ill-prepared for regulatory emissions reporting, while American and international companies cite confidence in climate risk disclosure.
4. Climate change practitioners support U.S. carbon pricing, yet are concerned about the level of public understanding on climate issues.
5. The carbon management software market is still in an embryonic stage.
6. Practitioners are concerned with peer competency; auditors are divided over the quality of work.
7. Carbon markets are not up to snuff; auditing needs enhanced governance.
8. GHG personnel are failing to meet current market requirements; competency concerns loom with the expansion of climate programs.
9. Climate employers and job seekers cite challenges in demonstrating and assessing carbon competency; they see professional certification as a fix.

The second “finding” reminds me of an opening scene in the John Wayne movie Donovan’s Reef. The CEO of a Boston shipping company was asking a board member, a prim, aged relative, his opinion on the matter before them. “No comment,” he said, “With reservations.” Package that quote up in fancy dress, and you have the 2010 Greenhouse Gas & Climate Change Workforce Needs Assessment Survey.

Cornell’s Cancun Climate Conference Crew

“A delegation of Cornell researchers will join the fight against climate change Monday in the annual United Nations Climate Change Conference in Cancun, Mexico.” So begins an article in the Cornell Daily Sun, the university’s student-run paper.

To repeat: this is a student-run paper and every allowance must be made for the immaturity, inexperience, and typical tendency toward enthusiasms of its writers. One does not expect nuance, or even complete correctness, in its coverage of major political events. This granted, this article encapsulates everything that is wrong with the public debate over global warming.

Or “Global Climate Change” as Schindler, the writer, put it. The event is of such moment to deserve full capitals, as if it were a personage. One cannot “fight” a common noun, but the perpetually concerned can do battle with a proper one.

Who’s going? Not I. I am only semi-attached to this august institution, so perhaps it is natural and right that my invitation has gone missing. However, no other person directly associated with matters atmospheric made the cut. Maybe they weren’t asked; or maybe they were, but had clearer heads.

We do know that the “Cornell Center for Sustainable Future — the Atkinson Center — has been working on coordinating the delegation.” You’ve got to hand it to activists: they picked the right word in sustainable. Who could be against sustainability? The amorphous nature of its definition is a tremendous advantageous in debate. Fling the word at an opponent, and he is immediately set back on his heels, too busy fending off suspicions that his motivations are purely pecuniary.

Anyway, on the sustainable Cornell bus will be “Eight undergraduates and ten graduate students” and three bosses. These are Antonio Bento, professor of Applied Economics and Management, Johannes Lehmann, professor of Soil Sciences, and Sean Sweeney, director of Cornell’s Global Labor Institute.

Bento and team “will present a theoretical and computational model of a cap-and-trade model”, which—do I need to say this?—is based on output from climate models. A model of a model of a model. Put another way: an approximation of a surmise of a guess. What could go wrong?

Sweeney—who I say truly is the man you want on your side in a negotiation—“will give a presentation on labor unions’ role in fighting climate change.” He said, “The labor unions are divided politically. There are those who see climate protection as a threat — the carbon-intensive industry.” A fascinating battle shaping up there. Will Sweeney be the one to break it to Teamsters members that their livelihood is harmful to the environment?

With lots of money to be had, it was only natural that organizations of every stripe, including unions, would want to nose around the trough. “About 200 labor organizations will attend the COP 16 and they will also have their own conference.” Mark that: their own conference, not the official one. Just being near all that cash is its own reward.

Lehmann, in a departure from the norm, will offer the meeting some research, “on how to avoid carbon dioxide losses from soils that would contribute to global warming, and how to increase organic carbon in soils that will be a sink for atmospheric carbon dioxide.” Could be interesting, that.

But if you begin to muse on soil physics, you’ll have missed the meat, which is that Lehmann, the only scientist in the group, is being shunted off onto a “side event”, which is “meant to inform the delegates.” Lehmann said, “The presentations by scientists are attended by negotiators that will hopefully be better informed through the material. Often, negotiators are directly interacting with presenters to deepen their knowledge.”

Do you see? This is the “Aha!” moment, dear readers. If you didn’t have it, reread that quotation, and pause at the word “Often.” The revelation is not even that the word is not “Always”. The key is that this meeting is being run by politicians who, if they care too, which oftentimes they do not, have to go out of their way to meet with actual scientists at what is, after all, purportedly a meeting about science.

Since physics has been relegated to “side events”, it must mean that the politicos have already made up their minds. But about what? I differ in many from thinking that these land sharks have grasped the fundamentals and uncertainties of thermodynamics and are convinced “It’s worse than we thought.” Instead, they are going for the same reason the unions are: power and money.

Older posts Newer posts

© 2014 William M. Briggs

Theme by Anders NorenUp ↑