IMS: Citation Indexes Stink

The Institute of Mathematical Statistics (I am a member) has issued a report on the wide-spread misuse of Citation Statistics.

The full report may be found here.

The non-surprising main findings are:

  • Statistics are not more accurate when they are improperly used; statistics can mislead when they are misused or misunderstood.
  • The objectivity of citations is illusory because the meaning of citations is not well-understood. A citation’s meaning can be very far from “impact”.
  • While having a single number to judge quality is indeed simple, it can lead to a shallow understanding of something as complicated as research. Numbers are not inherently superior to sound judgments.

The last point is not just relevant to citation statistics, but applies equally well to many areas, such as (thanks to Bernie for reminding me of this) trying to quantify “climate sensitivity” with just one number.

More findings from the report:

  • For journals, the impact factor is most often used for ranking. This is a simple average derived from the distribution of citations for a collection of articles in the journal. The average captures only a small amount of information about that distribution, and it is a rather crude statistic. In addition, there are many confounding factors when judging journals by citations, and any comparison of journals requires caution when using impact factors. Using the impact factor alone to judge a journal is like using weight alone to judge a person’s health.
  • For papers, instead of relying on the actual count of citations to compare individual papers, people frequently substitute the impact factor of the journals in which the papers appear. They believe that higher impact factors must mean higher citation counts. But this is often not the case! This is a pervasive misuse of statistics that needs to be challenged whenever and wherever it occurs.
  • For individual scientists, complete citation records can be difficult to compare. As a consequence, there have been attempts to find simple statistics that capture the full complexity of a scientist’s citation record with a single number. The most notable of these is the h?index, which seems to be gaining in popularity. But even a casual inspection of the h?index and its variants shows that these are naive attempts to understand complicated citation records. While they capture a small amount of information about the distribution of a scientist’s citations, they lose crucial information that is essential for the assessment of research.

I can report that many in medicine fixate and are enthralled by a journal’s “impact factor”, which is, as the report says, a horrible statistic—with an awful sounding name. The “h index” is “the largest n for which he/she has published n articles, each with at least n citations.”

Naturally, now that we statisticians have weighed in on the matter, we can expect a complete stoppage in the usage of citation statistics.

Variant on a theme

We, dear readers, have earlier dealt with the nonsensical argument, of which an Enlightened few are excessively fond, “There is no truth.” This is an argument often employed by those who embrace the idea that “all cultures are of equal value.” It is also commonly found, if sometimes not expressly stated, in academia in the “humanities”.

But the argument is ridiculously absurd and paradoxical, and in the same class as the 2600 year-old Epimenides paradox (Epimenides was a Cretan who said, “All Cretans are liars.”). A paradox, incidentally, is a man-made creation that stands in the way of a man-made theory gaining full acceptance. When a paradox arises, it implies, logically, that the theory that gave rise to it is flawed and should be modified or abandoned. But, usually, the theory is so beautiful or desirable that every possible effort is made to do away with the paradox (typically by calling it a “Problem” or ignoring it). The philosopher David Stove has brilliantly written about this in his book The Rationality of Induction.

Now, if we rationally believe the argument “There is no truth”, it must mean the argument is true. And if the argument is true, then the statement “there is no truth” is false because we at least believe the argument is true. Which of course means there is truth, so the argument is fallacious. Or nonsensical, actually. In other words, anybody who makes the argument and is convinced by it is making a grievous error or acting foolishly. This is bad news for those who theorize that human thought creates truth, or “truth” as they normally write it. From Stove again: writing “true” does not mean true, but only “believed to be true by so and so”, a definition as far from true as you can get.

Very well. Few actually utter the exact words “There is no truth”, probably because some internal B.S. detector senses something has gone awry. But there are, in common parlance, phrases which are entirely equivalent to “There is no truth.” Let’s look at one of them.

“Don’t be all judgmental”, is a phrase often heard immediately after you have pointed out that some behavior on the part of another was wrong or mistaken. Or it can be found in a simple example like this: you walk by a booth selling tie-dyed shirts and you say, “Those shirts are hideously ugly” and the booth owner says “Some people are so judgmental” which carries the implication that “being judgmental is wrong.”

The presupposition is that passing judgment on somebody’s “lifestyle” (for those who do not speak psychobabble, this means the English word behaviors) is an activity which is forbidden. It follows immediately that when the person says to you “Don’t be all judgmental” they are in fact passing judgment on your behavior. In other words, they are “being all judgmental.” It is, therefore, impossible not to pass judgment. I do not mean “impossible” in the colloquial sense of “unlikely”, but in the logical sense of “certainly cannot be no matter what.”

[UPDATE–thanks Nick!:] This is true whether tie-dyed shirts really are hideous or whether my comment was solicited (it was) or not, or whether the thought remains a thought and is forever unspoken. It might be, of course, that offering an unsolicited comment aloud is in poor taste, but it might also be that it is useful in the sense of discouraging aberrant behavior, such as that displayed by street vendors hawking ridiculous looking clothing.

So the next time somebody says to you “Don’t be all judgmental” you ask them “Aren’t you passing judgment on me?” Then get ready for a blank stare.

Ithaca update: hours and dogs as presidential candidates

The Ithaca Hours, mentioned in the previous post, quantify a barter system, trading “hours worked” at one task for equivalent “hours worked” at another. For example, you might trade one “hour” of “Cranio-sacral therapy, energy healing” for 10 hours of “Speaking & consulting on non-violent symbolic action.” Most services on offer are on the order of “Gentle Reiki energy sessions for health and growth” and ” movement coaching.” Some ordinary retail shops accept hours, but only for a small percentage of your overall bill. The Hours themselves have the logo “In Ithaca We Trust”, an expression the egotism of which I trust is obvious enough. The hours are, naturally, printed on hemp paper.

As I understand it, and here I might be off, trade, even though conducted in “Hours”, must still be ultimately accounted for in green-backs for tax purpose. “Hours” received are treated as ordinary income. Which, if true, makes the system truly worthless. But enlightened, and certainly enjoyable because, as their website says, it’s “fun to get and use something other than dollars (remember how much you enjoyed or still enjoy using monopoly money).” Thus, spending “Hours” is a form of play, though I find it odd that they would tout the game Monopoly, which is a game that teaches and celebrates capitalism.

The Ithaca Festival was this weekend on the Commons. This is a typical summer outdoor festival with arts & crafts and music. I counted not less than four booths that featured tie-dyed clothing, perhaps the ugliest form of body covering ever invented.

I went into a t-shirt shop (to find for my number two son a shirt emblazoned with “Ithaca Gun”, a now-defunct company that was justly famous for their shotguns) and some middle-aged ladies were discussing the upcoming election. “I’d vote for a dog before I’d vote for a republican!” said one. “I’d vote for a parakeet before I’d vote for McCain,” said another. “I can’t see why anybody would ever vote for a republican,” quipped the last.

The only thing strange about these commonplace comments is that they imply that the democrat party, lacking candidates of substance, will soon nominate animals to their tickets.

But you must hate us!

I am in Ithaca, New York, teaching a short course at Cornell University. Have you ever visited Ithaca? It was once voted the “most enlightened city in America” by the far-left magazine Utne Reader. Plenty of Volvos with “Impeach Bush” bumper stickers on them, a score of Tibetan bead shops in a desolate downtown area called the Commons, a own home-grown currency called “Hours” which is supposed to be more politically correct than greenbacks, and so on.

I was in a popular bar called the Chapter House (fantastic beer selection) and met a gentleman from England who was at Cornell taking a course from a well-known labor educator. This gentleman’s flight back home was canceled because of a thunderstorm. He is a union organizer for the Transit Workers in London. We had a nice chat over a few beers.

The bartender found out that my new friend was from England and asked him, “You must hate us over there.” By “us” he meant “Americana.” My friend said “No, we generally like Americans.” The bartender refused to accept this. “But you must hate us. Look at everything we have done!” My friend’s reply: “I was happy to come here. America is a great place.”

(By “we”, I assume the bartender did not include himself.)

This went back and forth a few times, my friend even describing a trip to Walmart to buy inexpensive jeans. The bartender lost heart and gave up. I felt sorry for him. There was nobody around to confirm his feelings of inferiority or to show him that he was not hated as he hoped he would be.

So the next time you are in Ithaca, please stop and tell somebody how much you dislike them. It will be sure to cheer them up.