Statistics

Increasing Replication Of Un-Reproducibility In Science

Best science picture of the year, courtesy of the Wall Street Journal (and thanks to reader Dan Hughes):

Replication in Science

In just the last decade—ten mere years—the number of “peer-reviewed” journal articles nearly doubled, from just over a million a year to about two million. That’s 5,500 papers per day. Even more telling is the right-hand graph: there were only eight-thousand journals in 1970, swelling to nearly 32,000 last year—with no end in sight.

I lost track of the number of journals relevant to my own field, statistics and probability. Well over 100. And if you consider that statistics is used in a majority of the 32,000 journals, there is no way to sample more than a fraction of what’s published. There are so many papers, the flood is so huge, that it gives me the contradictory feeling that I shouldn’t read anything. I figure there’s so much that I’ll miss that I might as well miss the lot. (I stick to books now—and there’s too many of them, too.)

Nor should most people fear missing a subscription since most of what makes it way into print isn’t worth reading. This follows from the historically well substantiated truism that most creative works are not lasting. You can nearly always find what’s valuable through the grapevine anyway (or by going to arxiv.org).

One consequence of the increasing number of papers is the number of mistakes, cheats, and non-replicable results are finally starting to hurt. My own field is responsible for more than its share of the calumny.

The WSJ quotes Glenn Begley, vice president of research at Amgen: “‘More often than not, we are unable to reproduce findings’ published by researchers in journals.” The paper opines, “This is one of medicine’s dirty secrets: Most results, including those that appear in top-flight peer-reviewed journals, can’t be reproduced.” It isn’t just medicine, of course. And the secret isn’t so secret.

Severals reasons for the degradation are adduced: in the lead, journals’ preference for positive results. (Also see this post: Are Academic Papers Growing Worse?)

[A]cademic researchers rarely conduct experiments in a “blinded” manner. This makes it easier to cherry-pick statistical findings that support a positive result. In the quest for jobs and funding, especially in an era of economic malaise, the growing army of scientists need more successful experiments to their name, not failed ones.

As I have long maintained, and by now I hope also proved, as long as you’re willing to put in the labor, “statistical significance” can be yours. Any set of data can be tickled into producing wee p-values, thus guaranteeing you a paper, i.e. a positive result showing what you had hoped to be true was true. Let the other guy worry about reproducing what you’ve done. And worry they’re doing.

Take, for example, Pfizer’s and Medivation’s stab at turning a “25-year-old Russian cold medicine into an effective drug for Alzheimer’s disease.” Medivation boss David Hung said of the published, peer-reviewed work on the drug: “Statistically, the studies were very robust.” Alas, after much effort and even more money, reproduction of the academic work could not be had. Which is by now a common story.

There are plenty of mistakes, too:

[T]he journal Science partially retracted a 2009 paper linking a virus to chronic fatigue syndrome because several labs couldn’t replicate the published results. The partial retraction came after two of the 13 study authors went back to the blood samples they analyzed from chronic-fatigue patients and found they were contaminated.

And don’t forget plain old cheating. Andrew Ferguson reports on the case of social psychologist, Diederik Stapel of Tilburg University in the Netherlands, a very naughty beguiler. He was a rising star in academics—150 papers and counting!—mostly because his work confirmed the biases of his colleagues. Advertising makes women feel bad about themselves. White men are homophobic. Messiness induces racism. And on and on. Stapel had plenty of publishable p-values, but only because he made up his data whole cloth (a way of cheating not yet covered on this blog).

Ferguson’s main point was the credulity of reporters, who repeat every “scientific” press release as gospel. We’ve seen it here when reviewing reports that seeing briefly a small picture of the American flag turns people into raving Republicans, or that attending a 4th of July parade while young turns one into a—can you guess?

Confirmation bias is one of those diseases the other guy gets.

Update All see this issue of Science. Ben Santer has a paper which argues that the twentieth century temperature increase (starting when? ending when? how much? the increase same every year?) has been reproduced, but that any reports of cooling have been greatly exaggerated.

Update See this: Questionable research practices are rife in psychology, survey suggests. Quote: “Questionable research practices, including testing increasing numbers of participants until a result is found, are the ‘steroids of scientific competition, artificially enhancing performance’.” You’ll remember we outlined this way of cheating a few weeks back. “[A]larming results suggest that one in ten psychologists has falsified research data.”

Categories: Statistics

11 replies »

  1. Those charts from the Journal make it look as if the number of reproducible studies has likely remained constant, and all the new journals are padded with junk studies.

  2. When I was heavily involved in research I found considerable amusement from reading the Journal of Irreproducible Results. If you are not aware it, I warn you that it is poking fun at science with desrcriptions of trivial and worthless experiments written by promenent researchers. Like the Nobell awards, science is chiding inself for taking itself too seriously. Unfortumately, this article is deadly serious because it suggests that the funding sources for research are being used to promote myths. The research institutions that support shoddy work are underpressure to endorse poor research even when they may know that the work is shoddy. One result that always emerges from research is the need for more research. Every answer produces more questions that need to be funded to find another set of unanswered questions. Questions foster grants. So if the research is not fruitful, they make it up and get their croneys to endorse it in the peer review processes. One only needs to look at Pennsylvania State Universities’ willingness to turn their back on child abuse to maintain a football dynasty. Some other educational institutions have research dynasties to keep alive with an influx of grants. It is called publish or perish. We could probably save millions of dollars in research funding by weeding out these unethical researchers and journals. One obvious result from reading the Climategate E-mails is that a core of scientists tried to manipulate the science to fit the out come biases of the funding sources and organizers of the UN-IPPC report.

  3. It’s over for science. The Santers of the Climate gang and the establishment that supported it have played the biggest specific part in its demise, but the end started much before. After WWII, Bush (Vannevar, not George) instituted the modern era of the big government funding machine, setting us on the course that resulted in science becoming just another government funded special interest. But the end starts much before even that, with the parting of the ways between science and the humanities in the Romantic era. The Romantics felt that art and science needed each other — that they were simply different ways of describing nature. The poets use poetry; the natural philosphers use science. Humphry Davy wrote beautiful poetry as he experimented in electrochemistry. He knew Shelly, Keats, and Byron, who were interested in science.

    In the end science sold its soul. Faust knew all about it.

  4. It started with a paper in Nature … Nature!

    As the Earth warms, many species are likely to disappear, often because of changing disease dynamics. Here we show that a recent mass extinction associated with pathogen outbreaks is tied to global warming.
    http://www.nature.com/nature/journal/v439/n7073/abs/nature04246.html

    … repeated by National Geographic …

    Global warming may cause widespread amphibian extinctions by triggering lethal epidemics, a new study reports.
    http://news.nationalgeographic.com/news/2006/01/0112_060112_frog_climate.html

    … questioned by some skeptics …

    … the primary reason that the amphibians appear to be disappearing in Central and South America bears little relation to temperature changes, but more likely is caused by the anthropogenic introduction of a virulent pathogen into the amphibian’s environment during the past 10 to 20 years.
    http://www.worldclimatereport.com/index.php/2006/01/11/jumping-to-conclusions-frogs-global-warming-and-nature/

    … again …

    Amid all of the horrible environmental insults we inflict upon the planet’s biodiversity, Rabbs’ fringe-limbed treefrog did not disappear in the wild because of overharvest for food, pets, or science, and we cannot lay the blame on familiar threats such as deforestation, climate change or environmental pollution. The culprit was emerging infectious disease. Amphibian chytridiomycosis is capable of directly eradicating otherwise large and stable populations and directly causing extinction of species that are otherwise unthreatened.
    http://www.americanscientist.org/issues/pub/lessons-of-the-lost/2

    … and again.

    There’s a fungus that infects many kinds of amphibians. Some get wiped out entirely—but it’s harbored harmlessly by others, so it’s impossible to eradicate. Over a hundred species have disappeared in the last 20 years!
    http://johncarlosbaez.wordpress.com/2011/12/08/the-global-amphibian-crisis/

  5. Does it happen in physics? Chemistry? Irreproducibility due to “honest” mistake or error happens and is a normal phenomenon. Somebody stumbled on an unexpected and got excited instead of tinkering with the results a little longer. But pulling data out of thin air creating “reality”? Is it possible that it happens in biological sciences more often because there are no precise enough rules of those disciplines (as opposed to rules of physics that allow to verify a new claim on several levels)?
    I guess the Austrians and here Friedman were right — it is best to keep the government out of most things. A well meant initiative (funds made available for science) created over time a powerful lobby that will defend itself, and it doesn’t matter that the results (quality of research) is…… what it is. Did you observe that it is the private sector that notices and complains about the problem?

  6. an economist notes that you get what you pay for….if you need to publish in order to keep a salried post, then you will publish

Leave a Reply

Your email address will not be published. Required fields are marked *