Mistakes are embarrassing, and getting over-excited about a statistical anomaly is silly. But these things happen, and the answer to building public confidence in science is not to pretend that they don’t

Last week I wrote about a possible signal for exciting new physics, seen in data from the two big “general purpose detectors” at CERN’s Large Hadron Collider. I was trying as carefully as I could to steer a course between being over-excited and overly conservative. It is difficult to be sure that your judgement is correct in such cases, but it is important to try. After all to quote Richard Feynman, in science

The first principle is that you must not fool yourself — and you are the easiest person to fool

… and if you understate a result through caution, you might feel almost as foolish as if you got carried away and claimed too much.

Now in the case I discussed it is not so serious, because the uncertainties involved are mainly statistical – that is, random – and that kind of uncertainty gets reduced automatically if you collect more data. And there are more data on the way. So if we’re fools, we won’t be fooled for long either way, and in the meantime we have a duty to publish what we see.

But if there is a race on for priority, or if other sorts of uncertainty are involved, premature or incorrect results can indeed cause problems. A couple of days after my article, a thoughtful comment piece appeared in Nature discussing some of these issues in a more general context. Jan Conrad, an astroparticle physicist, claims that

The field has cried wolf too many times and lost credibility

and he worries that false discoveries are undermining public trust in science. He lists some dubious results which have caused a stir amongst physicists and the general public over the past couple of years, including the faster-than-light-neutrinos that weren’t, the primordial gravitational waves that are probably just dust, and several Dark Matter candidates which remain shrouded in uncertainty and contradiction.

His argument has some merit; in some cases there is an apparent rush to release, and especially to over-interpret, provisional and sometimes incorrect data. This is sometimes done because of rivalries and competition, the desire to stake a claim. Other times it is simply that those concerned have found themselves too easy to fool. Such things can definitely distort the priorities of scientists and those who fund them.

But there are cases where we just have to report what we see, and we have to accept that sometimes what we see is misleading, and it may take more data, more experiments, to correct that. To hide a result because it doesn’t fit the current theory would be bad science. As I said at the time regarding the false faster-than-light neutrinos, imagine the conspiracy claims if the data had been suppressed because it didn’t fit Einstein’s theory. Of course, everyone wishes that they had found the problem with the dodgy cable before publishing, but mistakes do happen.

I also disagree with the criticism Conrad makes of my own experiment, where he objects to the fact that the eventual discovery of the Higgs boson was

preceded by press releases of weak but suggestive indications.

Given the public interest in the story, which we should welcome, it is completely implausible to think we would have gone into purdah from 2009 to 2012 and announced the Higgs boson data only when it passed the “5 sigma” discovery criteria, as it did three years ago almost to the day. Even had we done so, we would simply have missed a great chance to share the excitement, and the reality, of how scientific evidence is accumulated and understood.

Conrad also complains that too much of the premature release of results occurs on the open preprint server, arXiv. Again, I disagree. It is true that papers on the arXiv are usually not peer-reviewed, at least initially. However, while peer review as a journal article helps, it is no guarantee of veracity, as many retractions in various journals have shown. And the price – in terms of delay, and money – is high. As Chad Orzel notes in a response in Forbes, it would seem to suit Nature’s interests to undermine the arXiv, which has been distributing physics results more quickly and cheaply than the journals for decades, and is very heavily used by astro- and particle-physicists in particular.

Orzel argues that physics (and by implication science in general) should be open and people should get used to results being provisional, with frequent updates and corrections. I agree. Somehow we have to get beyond the opposing extremes that science is either 100% certainly correct, or else just another point of view to which there are many equally valid alternatives.

If you want a working understanding of the universe, which gives you the best chance of health for you and your loved ones, a stable environment to live in and cool gadgets to play with, science is absolutely the best we can do. But that doesn’t mean it is infallible. Particle physicist Brian Cox, much more of a logical positivist than a postmodern relativist, went so far as to say¹

Science is never right

and he’s correct, in the sense that it is always provisional, and is never, or at least never should be, dogmatic.

This of course is where many fools, cranks and fraudsters go wrong. Dogma can be overturned by aggression, by revolution, in the end by a simple rejection. Scientific knowledge applies whether or not an individual chooses to accept it, and it is continually revised and improved by acquiring additional evidence. That is its strength, that should be the basis of public confidence in science, and in my opinion the more widely that is understood the better.

¹At about 3:08 minutes in.

Also at the Guardian.