I have never tried to formally classify citations but I have noticed that an article with >10 citations is frequently only building on the work of a few and very often those few are previous articles by the authors. It seems that most citations are what I call the “Marketing Section.” The author is saying, “I am working on this problem because it is important and here are a bunch of other people working in this general area.” Very few of those articles are actually work that is being built upon in the sense that the author needs results from those papers.

In fairness, this is something I’ve observed mainly in image and signal processing papers. Other areas may be different.

Anyway, with citation indices becoming important I can easily see reviewers insisting that citations to their or their students’ work be inserted on very thin bases. There will be publication strategies whereby you insert references the work of editors or known reviewers. Or important work will be divided into multiple articles at the expense of clarity. Oh wait…

Even if they were statistically sound, I don’t really see these metrics as a path to objective evaluation of technical work. It will just make it clear what your “number” is. I’m pretty sure that’s not a way to improve science or the careers of its practitioners. It will probably just formalize the “mediocrity.”

Amen.

Peer review, at best, tends to enforce conservative middle-of-the-road results, at worst, mediocrity.

But a citation index is no better.

In fact, given all of human history, the only answer seems to be time.

Briggs

My research is currently being assessed by an RAE panel. There are two people on the panel who I know fairly well but I don’t work in the same field as them these days; the one panel member who does work in my field takes a rather different approach and so will probably not rate my research highly.
Of course it is possible to pick holes in h factor statistics by dreaming up hypothetical scenarios of researchers A and B. And of course such indices need to be used carefully, and not across different disciplines, and not to the complete exclusion of peer review. But I would rather be judged on my h factor (which is about 15) than by the subjective opinions of a small panel, only one of whom has looked at only 4 of my papers (that is how the RAE works). The nice thing about citations is that automatically, a large worldwide panel of experts in exactly my field is rating my research by citing it.
Another concern is that the ‘powers that be’ in the field got to where they are by peer review, and may fear losing their power as the emphasis on peer review is reduced. Would the queen support abolishing the monarchy?!

Yes, citation indices stink, but so does peer review.

But on the more general point of a single measure, such an approach needs to be used very circumspectly and with a very clear understanding of the measure and its relationship to what you are interested in. For example, as a measure of research productivity I would think that such an index would be a very poor substitute for looking at the actual product. On the other hand, if you want to use it for comparing academic departments in the same discipline then there might be some benefit.

I always like to start the morning with a good laugh.

Sigh.