A semantic confusion is clouding one of the most talked-about issues in research. Scientists agree that there is a crisis in reproducibility, but they can’t agree on what ‘reproducibility’ means.
The muddle is hampering communication about the problem and efforts to address it, a meeting last week on improving the reproducibility of preclinical research was told.
Most scientists — at least, those in biomedical research — have the idea that reproducible findings are those that give generally consistent results across slight variations in experimental set-up, says Ferric Fang, a microbiologist at the University of Washington in Seattle. “Reproduction is taking the idea of a scientific project and showing that it is robust enough to survive various sorts of analysis,” he says. That is, that it supports an expectation, for example that ‘reproducible’ preclinical results are those worth taking forward to clinical trials.
“I don’t think we can define reproducibility across the board”
But Fang adds that other definitions of the term are in common use. A second one is much narrower: a finding is reproducible if another researcher gets the same results when doing exactly the same experiment. On this interpretation, a fragile experiment that works under certain conditions in the laboratory, but not in other contexts, is still ‘reproducible’. And a third definition holds that a reproducible experiment is merely one that has been published with a sufficiently complete description — such as detailed methods — for another scientist to repeat it.
All these definitions point to the various problems that plague research. Scientists don’t want experiments that are poorly documented or unreliable, or that don’t give similar findings when the methods are slightly tweaked. But all of these issues have at times been framed as issues of reproducibility. “Reproducibility is shorthand for a lot of problems,” Jon Lorsch, head of the National Institute of General Medical Sciences, told attendees at the meeting, which was held in Bethesda, Maryland.
Without a shared understanding of the term, it can be unclear how scientists should respond when told that someone is “unable to reproduce” results in their paper, adds Ulrich Dirnagl, a stroke researcher at the Charité Medical University in Berlin. Challenges to research should be more clearly explained, he says.