Today’s guest post is in the form of an extended email by Gerald E. Quindry, Ph.D., P.E. who noticed something peculiar about some official findings on methane measurements.
In many of your posts, you have commented on the use, and abuse, of statistics in publications. Here is an example on which you may wish to comment. The documents I discuss below describe an investigation into the quantity of methane released from natural gas pipelines on the island of Manhattan. Both the preliminary and extended reports can be accessed from the Internet at: PDF link.
I came across these reports while searching for better, inexpensive method to monitor methane gas concentrations in air. I am an environmental engineer, and I have designed methane mitigation and monitoring systems for buildings constructed in areas where methane gas intrusion is a potential problem in building design. It may surprise you to learn that construction in a considerable portion of my home area of Southern California faces this issue. If you’ve ever been a tourist in Los Angeles, and visited the La Brea Tar Pits, the gas bubbles you see in the pits are…methane!
What first set off my “BS meter” was the following, from Page 20 of the “Extended” report:
These data were as follows:
Methane Concentrations in Ground-Level Air
Upwind 1.92 ppm ±0.003 ppm (99.9999% Confidence Interval)
Downwind 2.165 ppm ±0.021 ppm (99.9999% Confidence Interval)
I can’t recall ever before seeing chemical data presented with a “six nines” confidence interval; certainly never from data containing the kind of spatial and temporal variability that would be present in these data. I leave it to you, the “Statistician to the Stars” to explain how the numbers can be misinterpreted.
My real heartburn with the studies was the subsequent use of these data. These concentration values were used to calculate the change in methane levels in the air as it moved across Manhattan. Subsequently, that change was used to estimate the amount of methane released from the natural gas distribution piping underlying the City. (Natural gas is 85-95 percent methane.)
But a number can be both very precise, but very inaccurate, at the same time?
There are many potential sources of error in the analysis performed by the study. For example, there are many other sources of methane in the city, and no accounting of these other sources was made. Automobile exhaust contains methane from incomplete combustion of fuel. Sewers emit methane from the anaerobic bacteria that flourish there. Landfills and trash piles emit methane. Humans and animals emit methane. I also am doubtful that the analytical method used in the study was specific only to methane.
Many other volatile chemicals are released into the atmosphere, at restaurants, gas stations, dry cleaners, and the like. Paint contains volatile organic chemicals that are released while the paint dries. It is unclear that the data collected during the study would not be contaminated by passing near a release point for one of these air contaminants. Finally, (and this gets even deeper into the report methodology) it is my opinion that their treatment of boundary conditions and vertical mixing introduces huge potential errors in estimating the quantity of methane released to the atmosphere.
In summary, driving around the City, taking thousands of measurements of gas concentrations at the surface of the congested city streets, and then doing the calculations presented in the paper seems to be a poor choice in an attempt to quantify methane leaks from buried pipes.
The question then arises, why do the study? In my opinion, it is simply a side-battle in the climate change propaganda war.
Natural gas production in the United States is rising dramatically, due to advancements in the technology for drilling and extraction. The resulting oversupply of gas has dramatically reduced the price. That, in turn, has made it much more difficult for alternative energy projects to be economically viable without large, continuing, and reliable government subsidies. That has generated the need to discredit natural gas from its status as an economic, clean, and reliable energy source.
To accomplish this, the reports are publicized in a press release, “Natural Gas Emissions Measured in Manhattan Showing No Advantage to Natural Gas: Two Reports” (PDF) which is then picked up by the press and discussed on activist web sites, such as “New Study Exposes How Natural Gas Isn’t the Clean Fossil Fuel It’s Hyped up to Be” and “No smell of gas – but is that really OK?”
In my opinion, this shouting from the extremes, practiced by both sides in the debate, is no way to develop the sound energy and environmental policy that we desperately need.
Back to Briggs
Skipping the bizarre frequentist interpretation of confidence intervals and instead thinking like Bayesians, the “Upwind 1.92 ppm ±0.003 ppm (99.9999% Confidence Interval)” means that there is only a one in a million chance of seeing an upwind methane value outside the bounds 1.897 to 1.923 ppm for air coming into Manhattan. That’s strange because on p. 4 of the original report “Open country” values of 1.787 and 2.484 ppm were found.
They needed that narrow interval for their theory, though, which shows how much methane was supposedly added to air as it passed over Manhattan. If the interval on the incoming air was large, as it apparently should be, they could not claim leaky pipes added “8.6 billion cubic feet per year” (with no plus or minus) of methane.
Strangely, the confidence they claim on p. 4 is even higher: “highly reliable, 99.9999% confidence intervals ± <1% (0.021 ppm).” That’s sure some kind of confidence, boy.
And yes, a number can be both very precise, but very inaccurate, at the same time. You could have a gauge which prints readouts to arbitrary precision but which isn’t calibrated (the gauge reads “6.48572” but the actual value is “42”).
But skip all that. The most important part of Quindry’s criticisms are the other sources of over-confidence which the report does not address (such as the car exhaust, etc.).