I once saw a reference letter that a Senior Physicist had written for a colleague I had worked with closely. The letter compared my colleague to me, along the lines of “X is as good as Jon at Y, but in addition is excellent at Z”.
I wasn’t best pleased by this.
X and I had worked as a team, and I felt it was quite clear that I brought Y to the table while X brought Z (hang in there on the variable substitution) and together we were a truly excellent team covering both Y and Z. Senior Physicist was talking out of their… ignorance.
Anyway, it’s not that I have brooded on this for ages (really I haven’t and both me and X are doing fine career-wise, thanks). But I do worry about references, especially now I am myself a moderately Senior Physicist so am asked to write them quite often. I was reminded of it by this good article from Tim Tait, in Physics Today, about the culture and comparisons of reference letter writing. If such things affect or interest you, I recommend it.
Life and Physics 
Well I am certainly interested and I agree with the article you point us to. I am pleased to say that (so far) no one has ever asked me to compare the person I was writing a reference for with other people directly, certainly I have never seen a named list to rank against! That seems quite unethical and I wouldn’t do it. I think references are a real minefield and I wonder what, if any, evidence there is that they are actually useful (other than to confirm employment details etc.) and not subject to too much personal bias and indeed prejudice.
I’ve been given a named list to compare to a few times, basically from US universities making tenure decisions. Makes me uncomfortable, and when I don’t think an appropriate comparison can be made, I say so. But I always worry about how they might interpret that, too. The whole implication that there is some unified “top physicist” scale on which we can all be ranked against each other is (a) repugnant and (b) false, IMO.
Totally agree! The only time recently I was asked for a reference from a US university (tenured position) they were very careful about asking for inappropriate information, indeed they sent me a very clear guidance document making it very clear what sort of information I should/shouldn’t provide.