Endangered Reproducibility

Truth in Science Publishing: A Personal Perspective, by Thomas C. Südhof
Published: August 26, 2016

https://doi.org/10.1371/journal.pbio.1002547

"The other pillar of scientific truth, reproducibility, means that another scientist can repeat an experiment and arrive at the same results or, conversely, show that the results are not reproducible. Just as for peer review, multiple problems increasingly imperil reproducibility. For example, it’s not uncommon for an initial high-profile study to report amazing results with a stunning conclusion. Then, when the experiments are repeated, only trends toward the same conclusion are observed with increasingly smaller effect sizes. This outcome neither contradicts nor confirms the original study but is a dead end, and the original paper is slowly forgotten. As discussed above, the problem is not that the initial paper is fraudulent, but that the results were “tweaked” or selected, or represented a statistical outlier, leading to a misleading conclusion.

A second emerging reproducibility problem is that many experiments are by design impossible to repeat. As formalized by Karl Popper [Keuth Herbert (Ed.): Popper Karl. Logik der Forschung. Akademie-Verlag, Berlin 2004, ISBN 3-05-004085-8.], scientific truth requires interpersonal reproducibility. Based on this postulate, any conclusion that cannot be falsified because the underlying experiment cannot be repeated in exactly the same way is not a scientific conclusion. Many current experiments are so complex that differences in outcome can always be attributed to differences in experimental conditions ... If an experiment depends on multiple variables that cannot be reliably held constant, the scientific community should not accept the conclusions from such an experiment as true or false. Such conclusions are simply non-scientific, even if based on an experiment."

Let us remember that the "falsifiability", according to the philosopher Karl Popper, defines the inherent testability of any scientific hypothesis.

The Philosophy of Karl Popper, by Herbert Keuth

Cambridge University Press, 2005 

Peer-reviewed

Why we can't trust academic journals to tell the scientific truth

Academic journals don’t select the research they publish on scientific rigour alone. So why aren’t academics taking to the streets about this?

https://www.theguardian.com/higher-education-network/2017/jun/06/why-we-cant-trust-academic-journals-to-tell-the-scientific-truth

by Julian Kirchherr, Assistant professor at the faculty of geosciences, Utrecht University, the Netherlands (keyword scholar: Energy infrastructure environment development sustainability).

" These high-impact journals demand novel and surprising results. Unsuccessful replications are generally considered dull, even though they make important contributions to scientific understanding. Indeed, 44% of scientists (https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0005738&type=printable) who carried out an unsuccessful replication are unable to publish it. I have personal experience of this: my unsuccessful replication of a highly cited study has just been rejected by a high-impact journal. ..."  Il fatto che le pubblicazioni siano legate alla carriera ha un risultato. I ricercatori cercano di pubblicare su riviste ad alto impact factor. E per pubblicare "Fourteen per cent of scientists claim to know a scientist who has fabricated entire datasets, and 72% say they know one who has indulged in other questionable research practices such as dropping selected data points to sharpen their results."

I giornali di cui si parla sono giornali peer-reviewed. Gli articoli sono sottoposti ad un processo di revisione. Il problema sollevato da Julian è il seguente. Ha provato a replicare uno studio molto citato e non c'è riuscito; non è però riuscito a veder pubblicati i suoi risultati. Ma a monte c'è il problema che uno studio è stato pubblicato, anche se ha qualcosa che non funziona.