Can we trust scientific papers?

Australia Broadcast Center

3 days ago

Robyn Williams: The Science Show on Radio National. Over in America, universities are facing massive cuts, the loss of scientific staff and misinformation in the press, as well as on the dreaded social media. So what can we make of some of that? Here's Len Fisher who writes in the Blue Mountains and does research in Bristol.
Len Fisher: How do you know when to trust a scientist? For that matter, how does a scientist know when to trust a scientist? These questions are stimulated by an article that Robyn has brought to my attention. James Heathers from the Linnaeus University in Sweden claims that up to one scientific paper in seven may be at least partly fake and not to be trusted. So what is going wrong? Is Heathers paper itself partly fake? If not, what can we do to protect both science and the wider community from such fakery?
Let's start at the beginning. The trust of scientists in published science relies, or at least it used to rely, on peer review. Careful and critical examination of the work by one or more experienced fellow scientists who may award a pass, fail, or go back and try harder.
One problem with this process is the possibility of favouritism or even delaying the publication of a paper so that the reviewer, who will sometimes be working along similar lines, can get there first. I've seen it happen, although it does seem to be rare.
A more likely reason for delay is that reviewers are not paid for their work, and with many pressures on their time, reviewing tends to be put on the back burner. This gets to be real problem in a rapidly moving field, where younger workers are often involved and where priority can be important both for self-esteem and for actual promotion or the awarding of a degree. To get round the problem, authors these days tend to resort to online publication prior to peer review and conventional journal publication.
In fact, Heather's paper itself was published in this way, in what is called the Open Science Framework. It has not yet been peer reviewed. Does this matter for trust? At least Heather's paper is out there in the open, subject to critical commentary.
Some of that commentary has been pretty devastating. One fellow scientist, an expert in the sort of meta-analysis that Heathers uses, claims that Heathers' falsely labels studies with some problem as definitely being fake and incorrectly lumps together different studies measuring different phenomena. To his credit, Heathers accepts these criticisms in essence and even admits that his work is wildly non-systematic averaging as it does the results of 12 different studies of fakery in different fields and using different criteria. But he says that this is the best that he can do with the data available.
So far, so what? This is a squabble between scientists, and let's leave them to it. But the real problems start when the work impinges on the wider world. Scientific misconduct undermines public trust in science. It is rightly a matter of public concern.
So studies like those of Heathers find their way into the media, but with no mention of the essential caveats that, in Heathers' case, could well have meant that it would never have passed peer review. Even with quite careful journalistic treatment of the caveats, the majority of people who come across the story are likely to notice only the headline. 'One in seven science papers are fake'. Not even one in seven science papers could be fake.
Such headlines provide a convenient reason to reject scientific findings for those whose beliefs are challenged by these findings. A prime current example is the political dismissal of even very rigorous studies on the role of fossil fuels in global warming.
Let us return to the underlying problem, which is the current erosion of trust in science. What can be done to restore and maintain trust, both of scientists with each other and of the wider public the science?
The first thing to note is that most published papers are essentially trivial, small studies that receive at most one or two citations and whose results are of little concern except to the authors, whose quota of publications has increased, and their small circle of specialists. When a paper does address an important issue or reveal an important new finding that is of wider interest, then other scientists are likely either to repeat or to use the results. If there is fakery or sloppy science involved in the original publication, then hopefully the truth will out.
Unfortunately, this process can take some time, especially if the original work was convincing and fitted scientific preconceptions, as happened with the attribution of Alzheimer's disease to the presence of protein plaques in the spaces between brain cells. Also, replication can be difficult, or even impossible when specialist techniques sometimes possessed only by the originator are involved, or when conclusions are based on large-scale surveys that may have had inadequate controls. There is also the very human problem that scientists are likelier to accept the results of their fellows, especially the senior ones, without going to the bother of replication. This can be a problem of money or resources. Whatever the reason, it is a systemic problem in some disciplines witness the current well-publicised crisis of reproducibility in the psychology literature.
In general, however, replication, or even just its possibility, remains a powerful tool for the evocation of trust among scientists. The wonderful online retraction watch reports such exposures on a routine basis. Another welcome development is that of online pre-publication facilities such as arXiv which has the very peculiar spelling of small a small r capital X small i small v, which filters papers on the basis of the established reputations of the authors or recommendation by an established scientist. In fact, as I sit writing this talk, my collaborator on another publication, the Swedish polymath Anders Sandberg, is sitting in the corner of the same room submitting a paper of ours to Archive Physics but they won't accept it until it receives the approval of another known expert in the field, and even then it will be subject to moderation.
It is a pity that this same process can't be applied to books, especially those that make exaggerated claims based on little or no evidence. A recent New Scientist article has pointed out how few of such books are fact-checked. This may be just as well for the profits of the publishers, who often rely on the sales of such books.
The New Scientist itself employs two levels of fact-checking. But facts are dull, and sadly they are often trumped by simplistic exaggerations, especially when these appeal to pre-established prejudices. Maybe the answer is for scientists to learn to share more fully the very real excitement that comes from the pursuit and capture of facts, and their sharing with other scientists in an established atmosphere of trust. Only then will public trust in science already high in most countries, grow to overcome misleading stories about its very human vicissitudes.
Robyn Williams: Len Fisher is a Fellow of the Royal Society of New South Wales.