This article on the arXiv today seems interesting. I like this kind of meta-analysis. But two thing leap out at me.

Firstly, it focusses on searches for physics beyond the Standard Model. All such searches have come up with null results so far, and so they set exclusion limits on various hypotheses. I would like to see a similar analysis carried out for measurements of cross-sections, where we have predictions from the Standard Model to compare to, so we can see if the uncertainty estimates make sense there too. This might be trickier, since the correlations in theoretical uncertainties are typically even more difficult to quantify* than the mix of systematic uncertainties in searches (which also contain a theory element, but often with some data-driven modification applied).

Secondly, we really ought to release our results more often in an easily data-mineable form, so people don’t have to resort to re-digitising images. As the authors say:

The solution (HEPData) exists, we just have to get into the habit of using it more. (And we need to support it with appropriate funds, of course.)

All that said, and some minor caveats in the paper aside, I was pleased and relieved to see the conclusion:

In a world where the scientific endeavour has experienced recent reproducibility issues, we are happy to see that the results from the field of experimental high energy physics appear to be sound.

* I’m thinking particularly of the renormalisation scale uncertainty which estimates in a very ad hoc way the possible impact of higher order corrections in the perturbation theory.