Skip to content
July 1, 2008 | 9 Comments

Wired’s theory: the end of theory

Chris Anderson, over at Wired magazine, has written an article called The End of Theory: The Data Deluge Makes the Scientific Method Obsolete.

Anderson, whose thesis is that we no longer need to think because computers filled with petabytes of data will do that for us, doesn’t appear to be arguing serious—he’s merely jerking people’s chains to see if he can get a rise out of them. It worked in my case.

Most of the paper was written, I am supposing, with the assistance of Google’s PR department. For example:

Google’s founding philosophy is that we don’t know why this page is better than that one: If the statistics of incoming links say it is, that’s good enough. No semantic or causal analysis is required.

He also quotes Peter Norvig, Google’s research director, who said, “All models are wrong, and increasingly you can succeed without them.”


The scientific method is built around testable hypotheses….The models are then tested, and experiments confirm or falsify theoretical models of how the world works…But faced with massive data, this approach to science ? hypothesize, model, test ? is becoming obsolete.

Part of what is wrong with this argument is a simple misconception of what the word “model” means. Google’s use of page links as indicators of popularity is a model. Somebody thought of it, tested it, found it made reasonable predictions (as judged by us visitors who repeatedly return to Google because we find its link suggestions useful), and thus became ensconced as the backbone of its rating model. It did not spring into existence simply by collecting a massive amount of data. A human still had to interact with that data and make sense of it.

Norvig’s statement, which is false, is typical of the sort of hyperbole commonly found among computer scientists. Whatever they are currently working on is just what is needed to save the world. For example, probability theory was relabeled “fuzzy logic” when computer scientists discovered that some things are more certain than others, and nonlinear regression were re-cast as mysterious “neural networks,” which aren’t merely “fit” with data, as happens in statistical models, instead they learn (cue the spooky music).

I will admit, though, that their marketing department is the best among the sciences. “Fuzzy logic” is absolutely a cool sounding name which beats the hell out of anything other fields have come up with. But maybe they do too well because computer scientists often fall into the trap of believing their own press. They seem to believe, along with most civilians, that because a prediction is made by a computer it is somehow better than if some guy made it. They are always forgetting that some guy had to first tell the computer what to say.

Telling the computer what to say, my dear readers, is called—drum roll—modeling. In other words, you cannot mix together data to find unknown relationships without creating some sort of scheme or algorithm, which are just fancy names for models.

Very well—there will always be models and some will be useful. But blind reliance on “sophisticated and powerful” algorithms is certain to lead to trouble. This is because these models are based upon classical statistical methods, like correlation (not always linear), where it is easy to show that it becomes certain to find spurious relationships in data as the size of that data grows. It is also true that the number of these false-signals grow at a fast clip. In other words, the more data you have, the easier it becomes to fool yourself.

Modern statistical methods, no matter how clever the algorithm, will not being salvation either. The simple fact is that increasing the size of the data increases the chance of making a mistake. No matter what, then, a human will always have to judge the result, not only in and of itself, but how it fits in with what is known in other areas.

Incidentally, Anderson begins his article with the hackneyed, and false, paraphrase from George Box “All models are wrong, but some are useful.” It is easy to see that this statement is false. If I give you only this evidence: I will throw a die which has six sides, and just one side labeled ‘6’, the probability I see a ‘6’ is 1/6. That probability is a model of the outcome. Further, it is the correct model.

June 30, 2008 | 6 Comments

IMS: Citation Indexes Stink

The Institute of Mathematical Statistics (I am a member) has issued a report on the wide-spread misuse of Citation Statistics.

The full report may be found here.

The non-surprising main findings are:

  • Statistics are not more accurate when they are improperly used; statistics can mislead when they are misused or misunderstood.
  • The objectivity of citations is illusory because the meaning of citations is not well-understood. A citation’s meaning can be very far from “impact”.
  • While having a single number to judge quality is indeed simple, it can lead to a shallow understanding of something as complicated as research. Numbers are not inherently superior to sound judgments.

The last point is not just relevant to citation statistics, but applies equally well to many areas, such as (thanks to Bernie for reminding me of this) trying to quantify “climate sensitivity” with just one number.

More findings from the report:

  • For journals, the impact factor is most often used for ranking. This is a simple average derived from the distribution of citations for a collection of articles in the journal. The average captures only a small amount of information about that distribution, and it is a rather crude statistic. In addition, there are many confounding factors when judging journals by citations, and any comparison of journals requires caution when using impact factors. Using the impact factor alone to judge a journal is like using weight alone to judge a person’s health.
  • For papers, instead of relying on the actual count of citations to compare individual papers, people frequently substitute the impact factor of the journals in which the papers appear. They believe that higher impact factors must mean higher citation counts. But this is often not the case! This is a pervasive misuse of statistics that needs to be challenged whenever and wherever it occurs.
  • For individual scientists, complete citation records can be difficult to compare. As a consequence, there have been attempts to find simple statistics that capture the full complexity of a scientist’s citation record with a single number. The most notable of these is the h?index, which seems to be gaining in popularity. But even a casual inspection of the h?index and its variants shows that these are naive attempts to understand complicated citation records. While they capture a small amount of information about the distribution of a scientist’s citations, they lose crucial information that is essential for the assessment of research.

I can report that many in medicine fixate and are enthralled by a journal’s “impact factor”, which is, as the report says, a horrible statistic—with an awful sounding name. The “h index” is “the largest n for which he/she has published n articles, each with at least n citations.”

Naturally, now that we statisticians have weighed in on the matter, we can expect a complete stoppage in the usage of citation statistics.

June 24, 2008 | 6 Comments

Variant on a theme

We, dear readers, have earlier dealt with the nonsensical argument, of which an Enlightened few are excessively fond, “There is no truth.” This is an argument often employed by those who embrace the idea that “all cultures are of equal value.” It is also commonly found, if sometimes not expressly stated, in academia in the “humanities”.

But the argument is ridiculously absurd and paradoxical, and in the same class as the 2600 year-old Epimenides paradox (Epimenides was a Cretan who said, “All Cretans are liars.”). A paradox, incidentally, is a man-made creation that stands in the way of a man-made theory gaining full acceptance. When a paradox arises, it implies, logically, that the theory that gave rise to it is flawed and should be modified or abandoned. But, usually, the theory is so beautiful or desirable that every possible effort is made to do away with the paradox (typically by calling it a “Problem” or ignoring it). The philosopher David Stove has brilliantly written about this in his book The Rationality of Induction.

Now, if we rationally believe the argument “There is no truth”, it must mean the argument is true. And if the argument is true, then the statement “there is no truth” is false because we at least believe the argument is true. Which of course means there is truth, so the argument is fallacious. Or nonsensical, actually. In other words, anybody who makes the argument and is convinced by it is making a grievous error or acting foolishly. This is bad news for those who theorize that human thought creates truth, or “truth” as they normally write it. From Stove again: writing “true” does not mean true, but only “believed to be true by so and so”, a definition as far from true as you can get.

Very well. Few actually utter the exact words “There is no truth”, probably because some internal B.S. detector senses something has gone awry. But there are, in common parlance, phrases which are entirely equivalent to “There is no truth.” Let’s look at one of them.

“Don’t be all judgmental”, is a phrase often heard immediately after you have pointed out that some behavior on the part of another was wrong or mistaken. Or it can be found in a simple example like this: you walk by a booth selling tie-dyed shirts and you say, “Those shirts are hideously ugly” and the booth owner says “Some people are so judgmental” which carries the implication that “being judgmental is wrong.”

The presupposition is that passing judgment on somebody’s “lifestyle” (for those who do not speak psychobabble, this means the English word behaviors) is an activity which is forbidden. It follows immediately that when the person says to you “Don’t be all judgmental” they are in fact passing judgment on your behavior. In other words, they are “being all judgmental.” It is, therefore, impossible not to pass judgment. I do not mean “impossible” in the colloquial sense of “unlikely”, but in the logical sense of “certainly cannot be no matter what.”

[UPDATE–thanks Nick!:] This is true whether tie-dyed shirts really are hideous or whether my comment was solicited (it was) or not, or whether the thought remains a thought and is forever unspoken. It might be, of course, that offering an unsolicited comment aloud is in poor taste, but it might also be that it is useful in the sense of discouraging aberrant behavior, such as that displayed by street vendors hawking ridiculous looking clothing.

So the next time somebody says to you “Don’t be all judgmental” you ask them “Aren’t you passing judgment on me?” Then get ready for a blank stare.

June 23, 2008 | 7 Comments

Ithaca update: hours and dogs as presidential candidates

The Ithaca Hours, mentioned in the previous post, quantify a barter system, trading “hours worked” at one task for equivalent “hours worked” at another. For example, you might trade one “hour” of “Cranio-sacral therapy, energy healing” for 10 hours of “Speaking & consulting on non-violent symbolic action.” Most services on offer are on the order of “Gentle Reiki energy sessions for health and growth” and ” movement coaching.” Some ordinary retail shops accept hours, but only for a small percentage of your overall bill. The Hours themselves have the logo “In Ithaca We Trust”, an expression the egotism of which I trust is obvious enough. The hours are, naturally, printed on hemp paper.

As I understand it, and here I might be off, trade, even though conducted in “Hours”, must still be ultimately accounted for in green-backs for tax purpose. “Hours” received are treated as ordinary income. Which, if true, makes the system truly worthless. But enlightened, and certainly enjoyable because, as their website says, it’s “fun to get and use something other than dollars (remember how much you enjoyed or still enjoy using monopoly money).” Thus, spending “Hours” is a form of play, though I find it odd that they would tout the game Monopoly, which is a game that teaches and celebrates capitalism.

The Ithaca Festival was this weekend on the Commons. This is a typical summer outdoor festival with arts & crafts and music. I counted not less than four booths that featured tie-dyed clothing, perhaps the ugliest form of body covering ever invented.

I went into a t-shirt shop (to find for my number two son a shirt emblazoned with “Ithaca Gun”, a now-defunct company that was justly famous for their shotguns) and some middle-aged ladies were discussing the upcoming election. “I’d vote for a dog before I’d vote for a republican!” said one. “I’d vote for a parakeet before I’d vote for McCain,” said another. “I can’t see why anybody would ever vote for a republican,” quipped the last.

The only thing strange about these commonplace comments is that they imply that the democrat party, lacking candidates of substance, will soon nominate animals to their tickets.