I don't mean to reject completely the idea that some people are better at philosophy than others. I am better than any of my students, if only because they are just starting out. I have had enough philosophical education and teaching experience to be qualified to judge what grades their work deserves, for instance. (Even though no two philosophy professors are likely to give exactly the same grades to every essay in any given stack of papers.) And there might be some geniuses so great that they are clearly the best philosophers of their age. This does not seem to be true right now, but it might have been true at some times. Still, the idea that we can compare ethicists with logicians, say, and objectively find a member of the former group to be better or worse than some particular member of the latter group in every case seems absurd. Even within the set of ethicists, who you regard as better is likely to depend on who you think is closer to the truth. It is possible to respect the knowledge and skill of someone with whom one disagrees, but it is easier when you agree. Utilitarians are likely to think more highly of Peter Singer, for instance, than non-utilitarians do. And presumably no one would think that we can sensibly rank philosophical positions. Any ranking of philosophers who have PhDs but are not as great as Kant or Plato is not going to be purely a matter of personal taste or conviction, but these factors will muddy the waters so much that the ranking would have little value.
And then there is the question of training. The best athletes and musicians do not always make the best teachers. The same is likely to be true of philosophers. And graduate school in philosophy is not really a matter of going through a period of training anyway. Most of the time you are on your own reading (or not), not being coached. You might well be put right on this point or that in a seminar or by a comment on a paper, but there isn't a lot of this, as I recall. You learn various arguments and positions, and you sharpen your ability to defend your own positions and attack others. But there isn't so much intense sharpening that a dull mind will become an expert in just a few years. A good adviser can help you a lot, but good philosophers need not be very attentive advisers. Mine was, but you couldn't tell that by reading her publications or looking at her CV.
Ranking philosophers seems a bit like ranking football players. You can't really compare a striker with a defender, and even if Ronaldo stands out from almost everyone else, the rest of the pack (as long as the pack is suitably defined) is much of a muchness. If players were ranked, the 13th ranked player would be unlikely to be significantly better than the 23rd player. Numbers like these are meaningless. To then think that if you study with the 13th best you will improve more than if you studied with the 23rd is to compound the mistake. Philosophy professors provide reading lists and a kind of model of how to do philosophy, but the demonstration might not work particularly well in every case.
Which is all to say that I think there is something very problematic about ranking graduate programs in philosophy. This has been said before, and perhaps is obvious. But it's funny how seriously rankings and pedigree are taken. If you graduated from a good program you must be good because of the great training you will have received. It's easy to think this way. I think it's very hard to justify it though.
I do think it's good that there is information about graduate programs available online. When I was looking for a place to do my PhD in the 1980s I asked an American friend, who named what he thought were the top ten universities in the USA (without special regard to their philosophy programs), and D. Z. Phillips, who told me about some people in the States who had recently done work that he admired (not all of whom taught at PhD-granting institutions). I decided not to apply to Illinois because I wasn't sure that Peter Winch would stay there. So I applied to the University of Virginia (because Cora Diamond was there) and Rutgers (because Rupert Read went there, and he surely knew what he was doing). It worked out for me, but I was pretty ignorant. I never even considered Pittsburgh, for instance, which would have been a good place given my interests. (Although I might not have got in, of course.) So I'm glad it's easier now for people to find out who is where, who does what, and, to some extent, which places are considered better than others. But quality, being so subjective, matters mostly because you don't want to go somewhere that guarantees you will never get a job. I know there are hardly any jobs to get any more, but that's no reason to shoot your prospects in the foot unwittingly. Which leads me to think that information about which places are especially good for certain types of interest and a placement ranking is all that's needed. If there must be rankings by overall quality or reputation, let there be as many as possible so that both what consensus there is and what wide disagreement there is are apparent.
I'm tempted not to bother posting this because it all seems obvious or at least familiar. What is interesting to me is the combination of just how obvious at least some of it is and just how often it is (or at least appears to be) ignored in practice.
UPDATE: This (a comment by Helen De Cruz) seems relevant to the above:
Unsurprisingly, the philosophers' beliefs (theism, atheism, agnosticism) predicted to a significant extent how strong they thought these arguments were. It's no surprise that philosophers who were theists thought the arguments for theism were strong, and that the arguments against theism were weak, and that the opposite pattern held for atheists. Correlations between religious belief and perceived strength of argument were quite strong, e.g., an r score of -.483 for the cosmological argument. If argument evaluation is objective, how can we explain these strong correlations?Perhaps there is some flaw in her work, and perhaps a similar pattern would not be found among ethicists or philosophers of language, say. That is, maybe philosophy of religion is a special case. But I suspect it isn't. And I suspect that her findings are significant. There might be widespread agreement about who the top one or two Kant scholars or people working in x-phi are, but a) objectively evaluating Kant or x-phi seems to be a hopeless task, and b) below the very top, differences in ability are likely to be so small that they are either invisible or easily obscured by subjective noise (pedigree, personal acquaintance, etc.).
If I think about Wittgenstein scholars I can easily think of several who are neither definitely in the top two nor merely part of an indistinguishable mass, so perhaps I'm exaggerating. But something like the following categories seem to exist: the one or two people who I imagine are the most sought after for anthologies, conferences, etc.; ten to twenty big names whom it would make no sense to try to rank against each other; a mass of other people who are more or less indistinguishable from one another in terms of objective merit; other people who work on Wittgenstein but are basically unknown, perhaps because they have not published anything yet. The second category (the top ten-to-twenty) might turn out to be larger than I think if I actually started naming names. Anyway, I think that just about every Wittgenstein scholar who teaches at a PhD-granting institution probably belongs in this category. And the best scholars are not necessarily the best teachers or advisers. So anyone wanting to study Wittgenstein (and the point goes for anything else you might want to study) would be better off looking at how Wittgenstein-heavy a department is than at how highly ranked its one Wittgenstein scholar is.