|
The THES Rankings and the Dawn of Global Higher Education Data Standards
November 9, 2007
Alex Usher, Vice President, Educational Policy Institute
You might have missed this in North America because both the Chronicle and Inside Higher Ed managed to ignore it completely, but elsewhere in the world today, everyone is talking about the release of The Times Higher Education Supplement annual top 200 World Universities list this morning.
There will no doubt be some eyebrows raised about the Anglophilia at the top of the list: not that all top 10 are English-speaking universities (arguably true), but rather that 4 of them are actually located in England (much more controversial). There will also certainly be some tut-tutting about how some tweaking of methodology has led to some big changes in the rankings: the downgrading in importance of foreign students as a measure of quality has hit the LSE and some Australian institutions hard, leading to some really dumb headlines Down Under from journalists who can't be bothered to understand methodology.
Most people in the rankings business think that the main problem with the Times is the opaque way it constructs its sample for its reputational rankings - a not-unimportant question given that reputation makes up 50% of the sample. Moreover, this year's switch from using raw reputation scores to using normalized Z-scores has really shaken things up at the top-end of the rankings by reducing the advantage held by really top universities - UBC for instance, is now functionally equivalent to Harvard in the Peer Review score, which, no disrespect to UBC, is ludicrous.
I'll be honest and say that at the moment the THES Rankings are an inferior product to the Shanghai Jiao Tong rankings. Any publication that genuinely thinks McGill is the 12th best university in the world - ahead of UPenn, Stanford, Berkeley, the University of Tokyo, and the Ecole Normale Superieur - has a screw loose (yes, I can say that - I'm an alumni). This doesn't really pass the fall-down-laughing-test, which is the simplest way of validating any set of rankings.
It would probably help, too, if the Times would do a little fact-checking on the answers institutions provide. Much of McGill's advantage over U of T (who came 45th this year), for instance, stems from the fact that their student/staff ratio is - allegedly - 4 times better that U of T's. This, to put it mildly, is fantasy. (Let's be clear - I'm not saying anyone's cheating. I'm just saying that in good faith, the two institutions may not be reporting quite the same numerators and denominators on this item.)
But enough quibbling around the edges here. The THES Rankings may be a flawed product at the moment, but I would argue that over the coming five years, it may revolutionize the entire way that we measure performance in higher education. And that's because, behind the scenes, the Times has actually done a lot of the work necessary to improve itself significantly.
Here's the central problem faced by the Times: apart from bibliometrics and scientometrics, there aren't really any indicators which institutions in different countries collect and report in a common way. You name it: student numbers, staff counts, budgets, research income, completion rates - not everybody counts them, and when they do they don't count them the same way.
Faced with this, what does one do? Well, in the short-term there is really only one option: invent your own metrics. And the easiest way to do this is through surveys on reputation, just as the THES has done. One can quibble with some of their methodology, of course, but constructing a half-decent sample in a single country is hard enough, let alone doing so globally. So let's give them a bit of a break and say they've done an OK job. Not great, but OK (although, for the record, they've gone and made things worse with this Z-score thing).
Now, if this were all the THES were doing, I think we'd have reason to dismiss them as being somewhat trivial. But it's not. It turns out for the last three years, the THES' data partner, Quacquarelli Symonds (QS), has actually been collecting far more data from institutions than it has been publishing, asking institutions questions about library holdings, budgets and so forth. In other words, they are starting to get consistent data on all those kinds of indicators that one takes for granted in North America and which in many places forms the core of what we think of as rankings criteria.
QS hasn't started publishing any of this data yet - partly because not everyone is filling in the data and partly because they aren't convinced that they have drawn the definitions tightly enough to ensure that data across institutions is truly consistent. But their strategy is brilliant - quietly, behind the scenes, they are through trial and error starting to create genuinely global institutional indicators. Once those are up and running, the reputational stuff can take a back seat and we can start to have much more interesting comparisons between institutions.
And that's just step one. Now imagine QS were to go a little further. Say it were to ask institutions to send their students a little survey - nothing fancy, just a few satisfaction questions, maybe a few experiential questions. Not all institutions would participate at first, of course, but some would and the results could be published alongside the rankings.
Then things would start to get interesting. One of the problems of using survey data in rankings is that one has to get everyone involved because every indicator has to apply to every institution. That's one of the reasons its impractical (even it were desirable) to use NSSE data in rankings. The Harvards of this world don't want to do NSSE because - let's be honest - they don't want themselves compared to the East Podunk State colleges of this world.
But they might want to compare themselves to the Oxfords and Cambridges of this world. One could imagine, then, a scenario where gradually, more and more universities agree to participate in the surveys, until finally nearly all of them - even the Harvards - are in and the results can go from being published alongside the rankings to actually being within them.
Sound farfetched? I don't think so. In Europe, the idea of using student surveys within rankings systems (albeit not ones that use the THES' league table format) is now widely accepted thanks to the CHE, and institutions in Germany, Switzerland and Holland are all involved in this effort. In Canada, of course, the Globe and Mail/EPI/Strategic Counsel University Report Card already uses this methodology. As this approach gains ground from the bottom up, QS is in a position to spread this approach from the top down. The result would be that we could probably meet in the middle about a decade from now.
|
