See also Chapter 3.6 of Smashing Physics.
I noticed the other day that suddenly more people were looking at my blog. So it goes – Wordpress give you lots of cool graphs to obsess over. However, that particular demon is a mere spotty imp compared to the citation-count ogre.
A citation is when one scientific publication refers to another. Counting how often your papers are cited is one way to see how much influence your work is having, and it’s very tempting to keep watching them.
CITE ME!
In particle physics this is trickier that some other fields. My most highy-cited papers are measurements of proton structure from ZEUS. These were important measurements, and I helped build and run the experiment, but I made no direct input to those papers. This is common practice in particle physics, for good reasons which I alluded to here. But still I have my babies, the papers in which I recognize my own words, plots and ideas, as well as the results of my experiment.
Even amongst these, the current top two are funny ones.
The top “paper” which I actually partly wrote and edited contains no real data and no original theoretical ideas, and is not even published in a journal. It’s an 1852-page tome containing preparatory studies for using the ATLAS detector. It is useful, and the fact that it is cited a lot shows the level of interest in ATLAS, so it’s fair in that sense.
Next down is a real paper from ZEUS. We reported the mass we measured when two types of particles produced in our collisions were combined; neutral kaons and protons. We did this because some other experiments had seen a bump in the neutron-charged kaon mass distribution, which might have been the first observation of a hadron made of five quarks. In the Standard Model, all the hadrons we know of are made of either one quark and one antiquark, or three quarks. If it really was a five quark thing – a pentaquark – this would mean
- big physics news, and
- there should be a similar bump in our mass distribution.
We indeed saw a bump, though it was not completely compelling statistically and not necessarily in exactly the right place. Anyway, we did our job, we reported what we saw, and this was during a flurry of excitement so we got cited lots. Sadly it looks like the pentaquark thing was a false alarm, though our bump may have been real, but something else less interesting. Anyway.
Further down are lots of papers I’m very pleased with which I may bang on about in future (I already did about one of them).
This illustrates how dangerous citation counts can be as an indicator of merit. I’d happily lose the top two papers before most of the next ten, because the next ten contain more data or more original ideas. They advance knowledge more.
I went to a talk at UCL by Andrew Gregory last week, where I was surprised to hear that the idea that the planets orbit the Sun dates back at least to ancient Greece (Aristarchus). In fact I am now reading Simon Singh’s “Big Bang“, which points out that even Copernicus’s work was ignored for many years. Copernicus and Aristarchus would have struggled for tenure and grants based on citation counts during their lifetimes.
Here, in Norway, I met a father of string theory.
Before reading the Big Bang (and with Terry Pratchett‘s Nation in between – renaissance geek that I am) I read Lee Smolin’s “sɔısʎɥd ɥʇıʍ ǝןqnoɹʇ ǝɥʇ“. This describes the failure of string theory to come up with verifiable (or even falsifiable) breakthroughs in our understanding of the universe. This is despite decades of effort and investment to the extent that string theorists are the dominant grouping of theoretical physicists in terms of faculty positions in the US, apparently. I’m no theorist, I have no direct way to judge many of Smolin’s claims. But even I can see that because of the size of this community, mutual citation means that a good string theory paper will get cited lots more than a good paper on some other perhaps more promising but less well-followed tack. And citations make careers. Who knows how many breakthroughs have been lost through this feedback effect?
None of this is to say citation counts (or string theory!) are worthless, and of course good scientists cite good work. The counts are useful, used carefully. But in the short term they can underplay breakthroughs, which might take years to create a new field of people around them to provide citations. I suppose it may be better to work in a patent office for a while if the alternative is to redirect your research in a citation-hunt to secure grants and be appointed to a faculty position.
Oh, and the jump in my blog traffic was due to a very perceptive choice by the Guardian’s Alok Jha. And, so far, a large majority of the visitors only looked at the front page and didn’t read single complete post. Thanks for making it this far 🙂
Life and Physics 
Pingback: Interesting elsewhere – 18 June 2010 | Public Strategist
Hi Jon, I have to admit that a check mine once in a while (usually after a few drinks!). In my defence it’s definitely less than once a year. From a vanity p.o.v. It’s damn annoying have a few stuck below the magic 50. Still, other fish to fry these days…..
Nice post.
Interesting you mentioned jobs bc I thought for experimentalists papers and citation counts weren’t considered as much in hiring. precisely bc of the large author list on papers making an individuals contribution impossible to track that way. For theorists it’s a very big issue.
As an outsider I’ve also been worried about the testability of string theory bc of the landscape. I’ve never read smolins book but wonder what the aim is though? I feel uncomfortable when people take scientific debates into the court of public opinion, if that’s what he’s doing?
Tim – I was hugely encouraged to see Robert Peston(!) say, before Osborne’s prebudget statement:
“One area regarded as strategically important for the UK’s long term economic prospects, support for science, will be largely protected.”
http://www.bbc.co.uk/blogs/thereporters/robertpeston/2010/05/squeeze_on_industrial_interven.html
He was proved right on that occasion. He’s well briefed in general I think. If he got that from the treasury/government, maybe we have got some kind of message through. WIll it survive the budget? Who knows? You are right that the pressure needs to be kept on…
I hope you are right James, but what we know about evolutionary phase spaces is that they are generally topologically knotted enough that random processes become trapped in local minima. For instance, famously, if you are an ‘intelligent designer’ you might note that giraffe with long necks are going to do well. You might think that some extra vertebrae would be a good engineering move. Unfortunately, because no-one is in a position to have that thought, what you actually have in giraffe is the same number of vertebrae but massively stretched and hence weakened. Which gives you one way to response to creationists. Incidentally, Aristotle was very clear on this; he has an ‘as if’ notion of teleology in nature.
So if that’s the case, and the truth is either conceptually remote from string theory or there is no feasible path through the phase to space between the two, random mutation wouldn’t get us there.
I can see the attraction of your ‘wild card’ funding point, but I think in a world of highly limited resources, it would be hard to spin. But one of the things I am amazed about in the current debate is why no-one is talking about the stupidity of cutting the science budget right now. Everyone knows that I am about the most pro-cuts of anyone, but science is I guess almost literally the last thing I would cut. I would suppress drugs in the NHS I might need first – because science pays.
People get overexcited over being cited. It is a crude feedback system and citations bear varying weights and implications. If you’re cited by a prestigious journal, it means one thing, if you’re cited by a prestigious university, it’s another. If you’re cited in someone else’s experiment, another thing entirely. And if you’re cited by spongebob Squarepants, your name have entered the heads of 6 year olds who will one day become scientists because of your thrilling statistical aberrations.
But real influence comes in other researchers vectoring their research because of something you’ve said, even if you’re long dead when it happens.
But citations give you the feedback and approval you crave, and lets others support you if supporting you supports their own research, funding, grants, tenure.
It’s like the story Woody Allen told of a brother who thought he was a chicken, but who he never took to the doctor because he needed the eggs. Eggs. Do you like your theories scrambled, over easy or sunny side up?
Absoultely, Paul. And it depends on the vector too. In fact a good way to get cited a lot is to publish something wrong about something important; if it’s interestingly and honestly wrong that advances science, but if its just stupid you can “vector” people down a blind alley and waste much time. And rack up those citations!
e.g. I wonder what Andrew Wakefield’s citation count is on his MMR bollocks, which caused untold distress to millions of parents, gave lots of kids measles, and wasted lots of good researchers’ time.
This is easy to deal with – you just throw in some random mutations. If you then get stuck down some evolutionary cul-de-sack like string theory one of the mutations will eventually come up trumps to get you out.
So along with the normal peer review mechanism for funding there should be some small but finite probability that a given funding request will be financed regardless of the outcome of peer review. Obviously the probability should also depend on how much is being asked for (if it’s only 50p then it has a high chance of getting randomly funded).
One of the things I need to read when I try to derive Transcendental Idealism from Quantum Mechanics…
Pingback: Tweets that mention Survival of those that Fit « Life and Physics -- Topsy.com
I saw in Jim Al-Khalili’s interesting ‘Atom’ series the other day something about Gell-Mann sitting on quarks as a theory until SLAC saw something consistent with it. Apparently he thought everyone would hate it because it was wacky and people had had enough of fundamental new breakthroughs to additional levels of complexity and just wanted everything to settle down. Then when SLAC data came through which was consistent with extra substructure, Gell Mann was ready with some theory.
This is as it should be of course, but the concern would be that if no data is ever going to come through for strings – or worse yet, if we won’t ever even know whether string theory is falsifiable or not – then it will never go anywhere or do anything.
But then the question is why is it so dominant? What criterion are the physicists using to judge its power?
Hadn’t heard that story Tim but I can believe it.
Smolin has some interesting things things to say about why string theory is so dominant. I would summarise them as conservatism and patronage by the generation that saw how the Standard Model triumphed (but not by all those who developed it, e.g. T’Hooft is very into alternatives to strings). People appointing people like themselves. But Smolin is quite nuanced and well informed (he’s worked on strings too although he’s clearly fed up with it now). Worth a read.