Archive for February, 2008
Today, another brief (in the sense of intellectual content) essay, as I’m still working on the Madrid talk, the Heartland conference is this weekend, and I have to, believe it or not, do some work my masters want.
William F. Buckley, Jr. has died, God rest his soul. He famously said, “I’d rather be governed by the first 2000 names in the Boston phone book than by the dons of Harvard.” I can’t usefully add to the praise of this great man that has begun appearing since his death two days ago, but I can say something interesting about this statement.
There are several grades of pine “2 by 4’s”, the studs that make up the walls and ceilings of your house. Superior grades are made for exterior walls, lesser grades are useful for external projects, such as temporary bracing. A carpenter would never think of using a lesser grade to build your roof’s trusses, for example. Now, if you were run into a Home Depot and grab the first pine studs you came to (along with the book How to Build a Wall), thinking you could construct a sturdy structure on your own, you might be right. But you’re more likely to be wrong. So you would not hesitate to call in an expert, like my old dad, to either advise you of the proper materials or to build the thing himself.
Building an entire house, or even just one wall, is not easy. It is a complicated task requiring familiarity with a great number of tools, knowledge of various building techniques and materials, and near memorization of the local building codes. But however intricate a carpenter’s task is, we can see that it is manageable. Taken step by step, we can predict to great accuracy exactly what will happen when we, say, cut a board a certain way and nail it to another. In this sense, carpentry is a simple system.
There is no shortage of activities like this: for example baking, auto mechanics, surgery, accounting, electronic engineering, and even statistics. Each of these diverse occupations are similar in the sense that when we are plying that trade, we can pull a lever and we usually or even certainly know which cog will engage and therefore what output to expect. That is, once one has become an expert in that field. If we are not an expert and we need the services of one of these trades, we reach for phone book and find somebody who knows what he’s doing.
But there are other areas which are not so predictable. One of these is governance, which is concerned with controlling and forecasting the activity and behavior of humans. As everybody knows, it is impossible to reliably project what even one person will do on a consistent basis, let alone say what a city or country full of people will be like in five years. Human interactions are horribly, unimaginably complex and chaotic, and impossible to consistently predict.
Of course, not everyone thinks so. There is an empirically-observed relationship that says the more institutionalized formal education a person has, the more likely it is that that person believes he can predict human behavior. We call these persons academics. These are the people who make statements (usually in peer-reviewed journals) like, “If we eliminate private property, then there will be exact income equality” and “We can’t let WalMart build a store in our town because WalMart is a corporation.” (I cleaned up the language a bit, since this is a PG-rated blog.)
It is true, and it is good, that everybody has opinions on political matters, but most people, those without the massive institutionalized formal education, are smart enough to realize the true value of their opinions. Not so the academics, who are usually in thrall to a theory whose tenets dictate that if you pull this one lever, this exact result will always obtain. Two examples, “If we impose a carbon tax, global warming will cease” and “If the U.S.A. dismantles its nuclear weapons, so too will the rest of the world, which will then be a safer place.”
Political and economic theories are strong stuff and even the worst of them is indestructible. No amount of evidence or argument can kill them because they can always find refuge among the tenured. The academics believe in these theories ardently and often argue that they should be given the chance—because they are so educated and we are not—to implement them. They think that—quite modestly of course–because they are so smart and expert, that they can decide what is best for those not as smart and expert. Their hero is Plato who desired a country run by philosophers, the best of the best thinkers. In other words, people like them.
The ordinary, uneducated man is more likely to just want to be left alone in most matters and would design his laws accordingly. He would in general opt for freedom over guardianship. He is street-smart enough to know that his decisions often have unanticipated outcomes, and is therefore less lofty in his goals. And this is why Buckley would choose people from the phone book rather the from the campus.
February 29th, 2008
The other (for U.S. readers) Times, the original one, has compiled a list of the top 50 eco blogs and paid me the generous compliment of including this blog.
Other, and more important sites, like Climate Audit and Climate Resistance are featured in the Skeptical Category, with sites like Climate Debate Daily and Dot Earthlisted in the News Category.
The editors at the paper said that “the blogosphere is, frankly, a scary place” and that the “sheer diversity of the groups is staggering.” They “spent countless hours, days and months scouring the web” and ask “Did we manage to find the best?” Visit the site and find out.
February 27th, 2008
CRITICAL ASSESSMENT OF CLIMATE CHANGE PREDICTIONS FROM A SCIENTIFIC PERSPECTIVE
The Real Academia de Ciencias Exactas, Físicas y Naturales of Spain and the Fundación Ramón Areces are sponsoring that conference in Madrid on 2-3 April. The symposium will begin with the IPCC assessment and then move to “Critical assessment taking into consideration past (Pleistocene to historical) climate change and the nature of chemical forcing, as well as the characteristics of physical and numerical models used, their potentials, limitations and uncertainties.”
I’ve been asked to speak on the “Robustness and uncertainties of climate change predictions.” I have a deadline of this Saturday to hand in my abstract, and a couple of weeks to hand in a paper and presentation. I’m trying to walk a line between showing too much statistics and too little. By that I mean math. I keep going back and forth on this, trying to decide the best way to present. I haven’t decided, but, hey, I have three days left, right? Whatever I come up with will eventually be posted here. In a day or two–after my abstract is finished–I’ll post the conference program.
Reporters at the New York Times
A copy of Monday’s Times was given to me. I don’t subscribe, by the way, since I have learned from that august publication that because I am a veteran of the U.S. Armed Forces, I might explode at any moment and start murdering anybody in sight. I’m not disagreeing with this, of course; I just don’t want to be reminded of it.
Anyway, a lead story in the Business section, written by somebody called Noam Cohen, started thusly, “Of the many landmarks along a journalist’s career, two are among those that stand out: winning an award and making the government back down” (emphasis mine).
And people wonder where journalist’s cynicism comes from?
This site, written by someone called Clander, has been making the rounds and is hilarious. Some examples. “#75 Threatening to Move to Canada” (a friend of mine did this after Bush “stole” his first election), “#65 Co-Ed Sports” (one of my favorites), and “#62 Knowing what’s best for poor people“.
That post tells us
White people spend a lot of time of worrying about poor people….They feel guilty and sad that poor people shop at Wal*Mart instead of Whole Foods, that they vote Republican instead of Democratic, that they go to Community College/get a job instead of studying art at a University…It is a poorly guarded secret that, deep down, white people believe if given money and education that all poor people would be EXACTLY like them. In fact, the only reason that poor people make the choices they do is because they have not been given the means to make the right choices and care about the right things…But it is ESSENTIAL that you reassert that poor people do not make decisions based on free will. That news could crush white people and their hope for the future.
The accompanying picture is priceless. What’s even better are the reader’s comments, particularly those, presumably white, people who take exception to Clander’s observations.
February 27th, 2008
I am, of course, a statistician. So perhaps it will seem unusual to you when I say I wish there were fewer statistics done. And by that I mean that I’d like to see less statistical modeling done. I am happy to have more data collected, but am far less sanguine about the proliferation of studies based on statistical methods.
There are lots of reasons for this, which I will detail from time to time, but one of the main ones is how easy it is to mislead yourself, particularly if you use statistical procedures in a cookbook fashion. It takes more than a recipe to make an eatable cake.
Among the worst offenders are methods like data mining, sometimes called knowledge discovery, neural networks, and other methods that “automatically” find “significant” relationships between sets of data. In theory, there is nothing wrong with any of these methods. They are not, by themselves, evil. But they become pernicious when used without a true understanding of the data and the possible causal relationships that exist.
However, these methods are in continuous use and are highly touted. An oft-quoted success of data mining was the time a grocery store noticed that unaccompanied men who bought diapers also bought beer. A relationship between data which, we are told, would have gone unnoticed were it not for “powerful computer models.”
I don’t want to appear too negative: these methods can work and they are often used wisely. They can uncover previously unsuspected relationships that can be confirmed or disconfirmed upon collecting new data. Things only go sour when this second step, verifying the relationships with independent data, is ignored. Unfortunately, the temptation to forgo the all-important second step is usually overwhelming. Pressures such as cost of collecting new data, the desire to publish quickly, an inflated sense of certainty, and so on, all contribute to this prematurity.
Stepwise
Stepwise regression is a procedure to find the “best” model to predict y given a set of x’s. The y might be the item most likely bought (like beer) given a set of possible explanatory variables x, like x1 sex, x2 total amount spent, x3 diapers purchased or not, and on and on. The y might instead be total amount spent at a mall, or the probability of defaulting on a loan, or any other response you want to predict. The possibilities for the explanatory variables, the x’s, are limited only to your imagination and ability to collect data.
A regression takes the y and tried to find a multi-dimensional straight line fit between itself and the x’s (e.g., a two-dimensional straight line is a plane). Not all of the x’s will be “statistically significant1“; those that are not are eliminated from the final equation. We only want to keep those x’s that are helpful in explaining y. In order to do that, we need to have some measure of model “goodness”. The best measure of model goodness is one which measures how well that model does predicting independent data, which is data that in no way was used to fit the model. But obviously, we do not always have such data at hand, so we need another measure. One that is often picked is the Akaike Information Criterion (AIC), which measures how well the model fits the data that was used to fit the model.
Confusing? You don’t actually need to know anything about the AIC other than that lower numbers are better. Besides, the computer does the work for you, so you never have to actually learn about the AIC. What happens is that many combinations of x’s are tried, one by one, an AIC is computed for that combination, and the combination that has the lowest AIC becomes the “best” model. For example, combination 1 might contain (x2, x17, x22), while combination 2 might contain (x1, x3). When the number of x’s is large, the number of possible combinations is huge, so some sort of automatic process is needed to find the best model.
A summary: all your data is fed into a computer, and you want to model a response based on a large number of possible explanatory variables. The computer sorts through all the possible combinations of these explanatory variables, rates them by a model goodness criterion, and picks the one that is best. What could go wrong?
To show you how easy it is to mislead yourself with stepwise procedures, I did the following simulation. I generated 100 observations for y’s and 50 x’s (each of 100 observations of course). All of the observations were just made up numbers, each giving no information about the other. There are no relationships between the x’s and the y2. The computer, then, should tell me that the best model is no model at all.
But here is what it found: the stepwise procedure gave me a best combination model with 7 out of the original 50 x’s. But only 4 of those x’s met the usually criterion for being kept in a model (explained below), so my final model is this one:
| explan. |
p-value |
Pr(beta x| data)>0 |
x7 |
0.0053 |
0.991 |
x21 |
0.046 |
0.976 |
x27 |
0.00045 |
0.996 |
x43 |
0.0063 |
0.996 |
In classical statistics, an explanatory variable is kept in the model if it has a p-value< 0.05. In Bayesian statistics, an explanatory variable is kept in the model when the probability of that variable (well, of its coefficient being non-zero) is larger than, say, 0.90. Don't worry if you don't understand what any of that means---just know this: this model would pass any test, classical or modern, as being good. The model even had an adjusted R2 of 0.26, which is considered excellent in many fields (like marketing or sociology; R2 is a number between 0 and 1, higher numbers are better).
Nobody, or very very few, would notice that this model is completely made up. The reason is that, in real life, each of these x’s would have a name attached to it. If, for example, y was the amount spent on travel in a year, then some x’s might be x7=”married or not”, x21=”number of kids”, and so on. It is just too easy to concoct a reasonable story after the fact to say, “Of course, x7 should be in the model: after all, married people take vacations differently than do single people.” You might even then go on to publish a paper in the Journal of Hospitality Trends showing “statistically significant” relationships between being married and travel model spent.
And you would be believed.
I wouldn’t believe you, however, until you showed me how your model performed on a set of new data, say from next year’s travel figures. But this is so rarely done that I have yet to run across an example of it. When was the last time anybody read an article in a sociological, psychological, etc., journal in which truly independent data is used to show how a previously built model performed well or failed? If any of my readers have seen this, please drop me a note: you will have made the equivalent of a cryptozoological find.
Incidentally, generating these spurious models is effortless. I didn’t go through 100s of simulations to find one that looked especially misleading. I did just one simulation. Using this stepwise procedure practically guarantees that you will find a “statistically significant” yet spurious model.
1I will explain this unfortunate term later.
2I first did a “univariate analysis” and only fed into the stepwise routine those x’s which singly had p-values < 0.1. This is done to ease the computational burden of checking all models by first eliminating those x’s which are unlikely to be “important.” This is also a distressingly common procedure.
Here is the simulation code, to be run in the free and open source R statistical software:
library(MASS); # need be run only once per session
n<-100;
y<-rnorm(n) # "random" response
X<-matrix(rnorm(n*n/2),n,n/2) # "random" x's
f<-0
for (i in 1:(n/2)){
# univariate analysis; f stores p-values
f[i]<-anova(lm(y~X[,i]))$Pr[1]
}
i<-(f<.1) # only keep x's with p-values < 0.1
w<-data.frame(y,X[,i]) # w is just those x's with p<0.1 and y
fit<-lm(y~.,data=w) # model object to feed into stepwise
fit.aic <- stepAIC(fit) # stepwise
summary(fit.aic) # final model summary
February 25th, 2008
You know how it is. It’s dinner time, but you’re tying to cut back on the red meat. So what do you do? That’s right. You reach for a big ol’ bag of vegetarian intestines:
Look carefully at the bag. Two things are striking. The first is obviously the pile, the loops and loops, of fake intestines. You ask yourself: how did they ever get them to look so lifelike? Chinese attention to detail!
The second, noted by the caption “The picture is for reference only”, are the two exquisite bottles of wine, which, as everybody knows, go perfectly with boiled intestine.
Many of you by now want to know where to find this delicacy. Go to the Hong Kong Supermarket, frozen food aisle, in Elmhurst, Queens, right off the R, V, or G subway lines. Only $2.45, an exceptional bargain.
February 23rd, 2008
A few weeks ago I speculated what would happen if human-caused significant global warming (AGW) turned out to be false. There might be a number of people who will refuse to give up on the idea, even though it is false, because their desire that AGW be true would be overwhelming.
I guessed that these people would slip into pseudoscience, and so would need to generate excuses why we have not yet seen the effects of AGW. One possibility was human-created dust (aerosols) blocking incoming solar radiation. Another was “bad data”: AGW is true, the earth really is warmer, but the data somehow are corrupted. And so on.
I failed to anticipate the most preposterous excuse of all. I came across it while browsing the excellent site Climate Debate Daily, which today linked to Coby Beck’s article “How to Talk to a Global Warming Sceptic“. Beck gives a list of arguments typically offered by “skeptics” and then attempts to refute them. Some of these refutations are good, and worth reading.
His attempt at rebutting the skeptical criticism “The Modelers Won’t Tell Us How Confident the Models Are” furnishes us with our pseudoscientific excuse. The skeptical objection is
There is no indication of how much confidence we should have in the models. How are we supposed to know if it is a serious prediction or just a wild guess?
and Beck’s retort is
There is indeed a lot of uncertainty in what the future will be, but this is not all because of an imperfect understanding of how the climate works. A large part of it is simply not knowing how the human race will react to this danger and/or how the world economy will develope. Since these factors control what emissions of CO2 will accumulate in the atmosphere, which in turn influences the temperature, there is really no way for a climate model to predict what the future will be.
This is as lovely a non sequitur as you’re ever likely to find. I can’t help but wonder if he blushed when he wrote it; I know I did when I read it. This excuse is absolutely bullet proof. I am in awe of it. There is no possible observation that can negate it. Whatever happens is a win for its believer. If the temperature goes up, the believer can say, “Our theories predicted this.” If the temperature goes down, the believer can say, “There was no way to know the future.”
What the believer in this statement is asking us to do, if it is not already apparent, is this: he wants you to believe that his prognostications are true because AGW is true, but he also wants you to believe that he should not be held accountable for his predictions should they fail because AGW is true. Thus, AGW is just true.
Beck knows he is on thin ice, because he quickly tries to get his readers to forget about climate forecasts and focus on “climate sensitivity”, which is some measure showing how the atmosphere reacts to CO2. Of course, whatever this number is estimated to be means absolutely nothing about, has no bearing on, is meaningless to, is completely different than, is irrelevant to the context of, the performance of actual forecasts.
It is also absurd to claim that we cannot know “how the human race will react” to climate change while (tacitly or openly) simultaneously calling for legislation whose purpose is to knowingly direct human reactions.
So, if AGW does turn out to be false, those who still wish to believe in it will have to work very hard to come up with an excuse better than Beck’s (whose work “has been endorsed by top climate scientists”). I am willing to bet that it cannot be done.
February 20th, 2008
The old saying that “You can prove anything using statistics” isn’t true. It is a lie, and a damned lie, at that. It is an ugly, vicious, scurrilous distortion, undoubtedly promulgated by the legion of college graduates who had to suffer, sitting mystified, through poorly taught Statistics 101 classes, and never understood or trusted what they were told.
But, you might be happy to hear, the statement is almost true and is false only because of a technicality having to do with the logical word prove. I will explain this later.1
Now, most statistics texts, even advanced ones, if they talk about this subject at all, tend to cover it in vague or embarrassed passages, preferring to quickly return to more familiar ground. So if you haven’t heard about most of what I’m going to tell you, it isn’t your fault.
Before we can get too far, we need some notation to help us out. We call the data we want to predict y, and if we have some ancillary data that can help us predict y, we call it x. These are just letters that we use as place-holders so we don’t have to write out the full names of the variables each time. Do not let yourself be confused by the use of letters as place-holders!
An example. Suppose we wanted to predict a person’s income. Then “a person’s income” becomes y. Every time you see y you should think “a person’s income”: clearly, y is easier to write. To help us predict income, we might have the sex of the person, their highest level of education, their field of study, and so on. All these predictor variables we call x: when you see x, think “sex”, “education”, etc.
The business of statistics is to find a relationship between the y and the x: this relationship is called a model, which is just a function (a mathematical grouping) of the data y and x. We write this as y = f(x), and it means, “The thing we want to know (y) is best represented as a combination, a function, of the data (x).” So, with more shorthand, we write a mathematical combination, a function of x, as f(x). Every time you see a statistic quoted, there is an explicit or implicit “f(x)“, a model, lurking somewhere in the background. Whenever you hear the term “Our results are statistically significant“, there is again some model that has been computed. Even just taking the mean implies a model of the data.
The problem is that usually the function f(x) is not known and must be estimated, guessed at in some manner, or logically deduced. But that is a very difficult thing to do, so nearly all of the time the mathematical skeleton, the framework, of f(x) is written down as if it were known. The f(x) is often chosen by custom or habit or because alternatives are unknown. Different people, with the same x and y, may choose different f(x). Only one of them, or none of them, can be right, they both cannot be.
It is important to understand that all results (like saying “statistically significant”, computing p-values, confidence or credible intervals) are conditional on the model that chosen being true. Since it is rarely certain that the model used was true, the eventual results are stated with a certainty that is too strong. As an example, suppose your statistical model allowed you to say that a certain proposition was true “at the 90% level.” But if you are only, say, 50% sure that the model you used is the correct one, then your proposition is only true “at the 45% level” not at the 90% level, which is, of course, an entirely different conclusion. And if you have no idea how certain your model is, then it follows that you have no idea how certain your proposition is. To emphasize: the uncertainty in choosing the model is almost never taken into consideration.
However, even if the framework, the f(x), is known (or assumed known), certain numerical constants, called parameters, are still needed to flesh out the model skeleton (if you’re fitting a normal distribution, these are the μ and σ^2 you might have heard of). These must be guessed, too. Generally, however, everybody knows that the model’s parameters must be estimated. What you might not know is that the uncertainty in guessing the parameter values also has to carry through to statements of certainty about data propositions. Unfortunately, this is also rarely done: most statistical procedures focus on making statements about the parameters and virtually ignore actual, observable data. This again means that people come away from these procedures with an inflated sense of certainty.
If you don’t understand all this, especially the last part about parameters, don’t worry: just try to keep in mind that two things happen: a function f(x) is guessed at, and the parameters, the numerical constants, that make this equation complete must also be guessed at. The uncertainty of performing both of these operations must be carried through to any conclusions you make, though, again, this is almost never done.
These facts have enormous and rarely considered consequences. For one, it means that nearly all statistics results that you see published are overly boastful. This is especially true in certain academic fields where the models are almost always picked as the result of habit, even enforced habit, as editors of peer-reviewed journals are suspicious of anything new. This is why—using medical journals as an example—one day you will see a headline that touts “Eating Broccoli Reduces Risk of Breast Cancer,” only to later read, “The Broccolis; They Do Nothing!” It’s just too easy to find results that are “statistically significant” if you ignore the model and parameter uncertainties.
These facts, shocking as they might be, are not quite the revelation we’re after. You might suppose that there is some data-driven procedure out there, known only to statisticians, that would let you find both the right model and the right way to characterize its parameters. It can’t be that hard to search for the overall best model!
It’s not only hard, but impossible, a fact which leads us to the dirty secret: For any set of y and x, there is no unconditionally unique model, nor is there any unconditionally unique way to represent uncertainty in the model’s parameters.
Let’s illustrate this with respect to a time series. Our data is still y, but there is no specific x, or explanatory data, except for the index, or time points (x = time 1, time 2, etc.), which of course are important in time series. All we have is the data and the time points (understand that these don’t have be clock-on-the-wall “time” points, just numbers in a sequence).
Suppose we observe this sequence of numbers (a time series)
y = 2, 4, 6, 8; with index x = 1, 2, 3, 4
Our task is to estimate a model y = f(x). One possibility is Model A
f(x) = 2x
which fits the data perfectly, because x = 1, 2, 3, 4 and 2x = 2, 4, 6, 8 which is exactly what y equals. The “2″ is the parameter of the model, which here we’ll assume we know with certainty.
But Model B is
f(x) = 2x |sin[(2x+1)π/2]|
which also fits the data perfectly (don’t worry if you can’t see this—trust me, it’s an exact fit; the “2″s, the “1″ and the “π” are all known-for-certain parameters).
Which of these two models should we use? Obviously, the better one; we just have to define what we mean by better. Which model is better? Well, using any—and I mean any—of the statistical model goodness-of-fit measures that have ever, or will ever, be invented, both are identically good. Both models explain all the data we have seen without error, after all.
There is a Model C, Model D, Model E, and so on and on forever, all of which will fit the observed data perfectly and so, in this sense, will be indistinguishable from one another.
What to do? You could, and even should, wait for more data to come in, data you did not use in any way to fit your models, and see how well your models predict these new data. Most times, this will soon tell you which model is superior, or if you are only considering one model, it will tell you if it is reasonable. This eminently common-sense procedure, sadly, is almost never done outside the “hard” sciences (and not all the time inside these areas; witness climate models). Since there are an infinite number of models that will predict your data perfectly, it is no great trick to find one of them (or to find one that fits well according to some conventional standard). We again find that published results will be too sure of themselves.
Suppose in our example the new data is y = 10, 12, 14: both Models A and B still fit perfectly. By now, you might be getting a little suspicious, and say to yourself, “Since both of these models flawlessly guess the observed data, it doesn’t matter which one we pick! They are equally good.” If your goal was solely prediction of new data, then I would agree with you. However, the purpose of models is rarely just raw prediction. Usually, we want to explain the data we have, too.
Models A and B have dramatically different explanations of the data: A has a simple story (”time times 2!”) and B a complex one. Models C, D, E, and so on, all too have different stories. You cannot just pick A via some “Occam’s razor2” argument; meaning A is best because it is “simpler”, because there is no guarantee that the simpler model is always the better model.
The mystery of the secret lies in the word “unconditional”, which was a necessary word in describing the secret. We can now see that there is no unconditionally unique model. But there might very well be a conditionally correct one. That is, the model that is unique, and therefore best, might be logically deducible given some set of premises that must be fulfilled. Suppose those premises were “The model must be linear and contain only one positive parameter,” then Model B is out and can no longer be considered. Model A is then our only choice: we do not, given these premises, even need to examine Models C, D, and so on, because Model A is the only function that fills the bill; we have logically deduced the form of Model A given these premises.
It is these necessary external premises that help us with the explanatory portion of the model. They are usually such that they demand the current model be consonant with other known models, or that the current model meet certain physical, biological, or mathematical expectations. Regardless, the premises are entirely external to the data at hand, and may themselves be the result of other logical arguments. Knowing the premises, and assuming they are sound and true, gives us our model.
The most common, unspoken of course, premise is loosely “The data must be described by a straight line and a normal distribution”, which, when invoked, describes the vast majority of classical statistical procedures (regression, correlation, ANOVA, and on and on). Which brings us full circle: the model and statements you make based on it are correct given the “straight line” premise is true, it is just that the “straight line” premise might be, and usually is, false.3
Because there are no unconditional criteria which can judge which statistical model is best, you often hear people making the most outrageous statistical claims, usually based upon some model that happened to “fit the data well.” Only, these claims are not proved, because to be “proved” means to be deduced with certainty given premises that are true, and conclusions based on statistical models can only ever be probable (less than certain and more than false). Therefore, when you read somebody’s results, pay less attention to the model they used and more to the list of premises (or reasons) given as to why that model is the best one so that you can estimate how likely the model that was used is true.
Since that is a difficult task, at least demand that the model be able to predict new data well: data that was not used, in any way, in developing the model. Unfortunately, if you added that criterion to the list of things required before a paper could be published, you would cause a drastic reduction in scholarly output in many fields (and we can’t have that, can we?).
1I really would like people to give me some feedback. This stuff is unbelievably complicated and it is a brutal struggle finding simple ways of explaining it. In future essays, I’ll give examples from real-life journal articles.
2Occam’s razor arguments are purely statistical and go, “In the past, most simple models turned out better than complex models; I can now choose either a simple or complex model; therefore, the simple model I now have is more likely to be better.”
3Why these “false” models sometimes “work” will be the discussion of another article; but, basically, it has to do with people changing the definition of what the model is mid-stream.
February 18th, 2008
Every now and then I have a glance at my log files to see what kinds of key words people type into sites like Google and who are subsequently directed to my site. It won’t surprise you that I see things like briggs and bad statistics examples. But there is a class of keywords that I can only describe as odd, even, at times, worrying. Here are those keywords (all spellings are as they were found), split into rough categories. My comments, if any, appear in parentheses. Each of these keywords are real.
Statistics
don't forget about us model (I could never)great statisticians (flatterer)how to exaggerate (think big, think big)i need to be statician (it can be a powerful force, it’s true; learning to spell it correctly will help)some pictures of statistician (here’s somebody with a lot of time on their hands)statisticians aviod doing things because other people are doing it (I think he has us confused with accountants)statistician god exists (His name is Stochastikos)virginity statistics (score: 0 to 0)lifelong virginity statistics (score still tied)what to look for in a statician (get one of the tall ones; we have a sense of humor)why do statisticians love tables? (because we can’t help ourselves)you cannot be a scientist if you are not a good mathematician (I have the feeling that this person desired a negative answer)
factors that cause zombism (blogging…)recorded zombie outbreakswhat year will zombies take over the earth? (has to be soon)wild zombies (as opposed to domesticated?)will zombie attacks happenzombies can happen (he might have been trying to answer the other guy)zombies in naturezombies true or false
Miscellaneous
800g balls (mine are only 760g–in petanque, of course!)
anything (I can see Google knows where to go…)
beer does not have enough alcohol (which is why I tend to stick with rum)
home is where the heart is william briggs (somebody’s trying to give me a lesson)
horizontal alcoholic (is there any other kind?)
how does pseudoscience effect the mind (badly)
lee majors george bush (you can’t go wrong aligning yourself with the six-million dollar man)
man's got his limits briggs (true enough; must be same advice giver as before)
purposely causing someone to get cancer (oh my…no murder tips here)
sentence with the word, "impossibility" (shouldn’t be hard to come by)
what can we do not to be poor (get a job)
February 17th, 2008
In 1914, there was a consensus among geologists that the earth under our feet was permanently fixed, and that it was absurd to think it could be otherwise. But in 1915, Alfred Wegener fought an enormous battle to convince them of the relevance of plate tectonics.
In 1904, there was a consensus among physicists that Newtonian mechanics was, at last, the final word in explaining the workings of the world. All that was left to do was to mop up the details. But in 1905, Einstein and a few others soon convinced them that this view was false.
In 1544, there was a consensus among mathematicians that it was impossible to calculate the square root of negative one, and that to even consider the operation was absurd. But in 1545, Cardano proved that, if you wanted to solve polynomial equations, then complex numbers were a necessity.
In 1972, there was a consensus among psychiatrists that homosexuality was a psychological, treatable, sickness. But in 1973, the American Psychiatric Association held court and voted for a new consensus to say that it was not.
In 1979, there was a consensus among paleontologists that the dinosaurs’ demise was a long, drawn out affair, lasting millions of years. But in 1980, Alvarez, father and son, introduced evidence of a cataclysmic cometary impact 65 million years before.
In 1858, there was a consensus among biologists that the animal species that surround us were put there as God designed them. But in 1859, the book On the Origin of Species appeared.
In 1928, there was a consensus among astronomers that the heavens were static, the boundaries of the universe constant. But in 1929, Hubble observed his red shift among the stars.
In 1834, there was a consensus among physicians that human disease was spontaneously occurring, due to imbalanced humours. But in 1835, Bassi and later Pasteur, introduced doctors to the germ theory.
All these are, obviously, but a small fraction of the historical examples of consensus in science, though I have tried to pick the events that were the most jarring and radical upsets. Here are two modern cases.
In 2008, there is a consensus among climatologists that mankind has and will cause irrevocable and dangerous changes to the Earth’s temperature.
In 2008, there is a consensus among physicists that most of nature’s physical dimensions are hidden away and can only be discovered mathematically, by the mechanisms of string theory.
In addition to the historical list, there are, just as obviously, equally many examples of consensus that turned out to be true. And, to be sure, even when the consensus view was false, it was often rational to believe it.
So I use these specimens only to show two things: (1) from the existence of a consensus, it does not follow that the claims of the consensus are true. (2) The chance that the consensus view turns out to be false is much larger than you would have thought.
These are not news, but they are facts that are often forgotten.
February 15th, 2008
This subject comes up so often and in so many places, and so many people ask me about it, that I thought a short explanation would be appropriate. You may also search for “running mean” (on this site) for more examples.
Specifically, several readers asked me to comment on this post at Climate Audit, in which appears an analysis whereby, loosely, two time series were smoothed and the correlation between them was computed. It was found that this correlation was large and, it was thought, significant.
I want to give you, what I hope is, a simple explanation of why you should not apply smoothing before taking correlation. What I don’t want to discuss is that if you do smooth first, you face the burden of carrying through the uncertainty of that smoothing to the estimated correlations, which will be far less certain than when computed for unsmoothed data. I mean, any classical statistical test you do on the smoothed correlations will give you p-values that are too small, confidence intervals too narrow, etc. In short, you can be easily misled.
Here is an easy way to think of it: Suppose you take 100 made-up numbers; the knowledge of any of them is irrelevant towards knowing the value of any of the others. The only thing we do know about these numbers is that we can describe our uncertainty in their values by using the standard normal distribution (the classical way to say this is “generate 100 random normals”). Call these numbers C. Take another set of “random normals” and call them T.
I hope everybody can see that the correlation between T and C will be close to 0. The theoretical value is 0, because, of course, the numbers are just made up. (I won’t talk about what correlation is or how to compute it here: but higher correlations mean that T and C are more related.)
The following explanation holds for any smoother and not just running means. Now let’s apply an “eight-year running mean” smoothing filter to both T and C. This means, roughly, take the 15th number in the T series and replace it by an average of the 8th and 9th and 10th and … and 15th. The idea is, that observation number 15 is “noisy” by itself, but we can “see it better” if we average out some of the noise. We obviously smooth each of the numbers and not just the 15th.
Don’t forget that we made these numbers up: if we take the mean of all the numbers in T and C we should get numbers close to 0 for both series; again, theoretically, the means are 0. Since each of the numbers, in either series, is independent of its neighbors, the smoothing will tend to bring the numbers closer to their actual mean. And the more “years” we take in our running mean, the closer each of the numbers will be to the overall mean of T and C.
Now let T' = 0,0,0,...,0 and C' = 0,0,0,...,0. What can we say about each of these series? They are identical, of course, and so are perfectly correlated. So any process which tends to take the original series T and C and make them look like T' and C' will tend to increase the correlation between them.
In other words, smoothing induces spurious correlations.
Technical notes: in classical statistics any attempt to calculate the ordinary correlation between T' and C' fails because that philosophy cannot compute an estimate of the standard deviation of each series. Again, any smoothing method will work this magic, not just running means. In order to “carry through” the uncertainty, you need a carefully described model of the smoother and the original series, fixing distributions for all parameters, etc. etc. The whole also works if T and C are time series; i.e. the individual values of each series are not independent. I’m sure I’ve forgotten something, but I’m sure that many polite readers will supply a list of my faults.
February 14th, 2008
Previous Posts
