Apparently it is university rankings season. Newspapers in Ireland were full of the news that in the latest QS rankings most Irish universities have slipped although UCC managed to put a positive spin on by some obscure claim to be Ireland’s only “5 star university”- despite being ranked below UCD & TCD.
I generally don’t get excited about these things one way or another although it is always amusing to see universities splash the good news on their websites but remain curiously silent when the news is bad. Such is human nature. It also seems to me a lot of the academics who comment on them are strangely uncritical of the methodology or take the line “oh yes we know they are not perfect but they matter anyway”. Economists, in particular, have no excuse as we know about the limitations of data. Well maybe theorists don’t.
With the latest QS rankings my attention was drawn to one of the criteria “International Faculty” i.e. the proportion of academics who are, I assume, foreign to the university’s country. Now it only gets 5% weighting so its not very important but that doesn’t explain its inclusion. This attempts to justify it. In my opinion, it is complete bullshit and I find it hard to see how any intelligent person familiar with universities could subscribe to that view (or indeed write so badly).
It is quite funny to see the claim that because the rankings are international that it is essential to include the % of international faculty. On the contrary, it is a reason not to. If you are in a big country, like the US, there is a bigger supply of your own nationals. If no other country speaks your language then there is a smaller supply. So what? People should be judged on their merits not on the colour of their passports.
I think serious universities should have the self-confidence to call a spade a spade and say we simply don’t care about these Mickey Mouse numbers (with apologies to Disney).
Pingback: The Irish Economy » Blog Archive » The best university in all the land
There are many ways to rank universities. None are perfect. Unfortunately, the worst methods attract most media attention. Rankings matter for students, funding, faculty. We’d better understand these rankings, whether we like them or not.
Pingback: Ninth Level Ireland » Blog Archive » On bullshit and university rankings
Richard, I dont deny they matter & this particular criterion is small. But it is important that we adopt the same scholarly rigour towards them as towards our own research so if we think it is nonsense, say so. How else will they be changed unless the sector points out the problems? I may be wrong, but my sense is that a lot of universities are somewhat cowed by it all. Again I maybe wrong but I suspect Harvard doesn’t care.
For a thorough ranking, see http://www.cwts.nl/ranking/LeidenRankingWebSite.html
You immediately see why that does not make headlines.
I was shocked when I looked at these rankings some time ago…carefully put together & explained, they kind of made sense. It will never catch on.
Right… Number of publications isn’t a bullshit metric. No, not at all. 4 publications by Einstein in 1905 are more than outweighed by 15 from Richard Tol.
Ernie,
That’s true for one individual (and Einstein was indeed some individual) but not so easy to dismiss over a very large number of people.
Kevin,
There is some history in people travelling abroad to work in a great department or with a great man (e.g. Cambridge physics in the 1920′s/’30′s) and it must be trying to capture that today. People have travelled to Ireland to work with Neary in UCD, etc., it does count for something, surely? But how to weigh it?…
Incidentally I understand travelling abroad for that purpose is also the reason beginning-academics try to attain the particular letters “Ph.D.”
In the U.S. in the 19th century if you wanted to be exposed to the best people / taken seriously / get a job, in mathematics, you had to study in Germany (Gauss etc.). They then took the (relatively new) “PhD” system, along with the seminar, back to the U.S. and introduced it there, from where it spread to Canada. At the end of the First World War it was imported into Ireland and Britain from the U.S. & Canada (prior to that, ‘Fellowship’ reigned) with changing fashions/’standardisation’, but also as a way for the Irish and British universities to extract fees from wealthy(ish) US students. TCD had introduced a 2 year research M.Sc. in ~1918 (I think), but there was not enough foreign take-up, so they added a year and substituted the magic letters “PhD”, and of course it has been there ever since.
See Webb & McDowell for more (the latter passed away last week, his obit was in Sat’s Irish Times)
Same idea at Oxford, but (as you well know) they decided to retain their “DPhil” for these new research degrees.
“Ernie” Hard cases make bad law so I am not sure that the Einstein example is that helpful. But if the point is that simply counting papers is potentially misleading then sure. It gives people an incentive to just churn stuff out. We all know that and good rankings will allow for that, either by some form of weighting (by citation) or by journal ranking. In economics, this can work reasonably well as there is pretty strong agreement about what the prestigious journals are. Of course it is not perfect but it is certainly better than nothing. My guess is that this would not be true in some disciplines where there may be big philosophical or other divides.
John: I don’t get your point at all. If people travel to Cambridge from India or Liverpool to work with X thats great but what does it matter which?
On a related note, it is worth considering how the UK’s Research Assessment Exercise (RAE) compares. While this exercise only relates to research, and of course only to the UK (and while noting the criticisms that have been made about it: http://en.wikipedia.org/wiki/Research_Assessment_Exercise#Criticism); there are some interesting papers by economists on the RAE, that are worth mentioning.
First, for the uninitiated, the Wiki entry cited above describes the RAE as: “an exercise undertaken approximately every 5 years on behalf of the four UK higher education funding councils (HEFCE, SHEFC, HEFCW, DELNI) to evaluate the quality of research undertaken by British higher education institutions. RAE submissions from each subject area (or unit of assessment) are given a rank by a subject specialist *peer review* panel. The rankings are used to inform the allocation of quality weighted research funding (QR) each higher education institution receives from their national funding council. Previous RAEs took place in 1986, 1989, 1992, 1996 and 2001. The most recent results were published in December 2008.”
Obviously enough, these rankings are extremely meaningful, in the sense that they mean a lot (re: funding) to each department (or unit of assessment) under scrutiny. But why not use bibliometric methods instead? Something along the lines of the Leiden Ranking system linked above?
From reading this paper on the REA (http://www.lums.lancs.ac.uk/publications/viewpdf/006236/) by Jim Taylor and Ian Walker (2009), it appears that the explanation is as follows:
“A fundamental criticism of using a journal quality index to assess the quality of a department’s research output is that not all publications in high quality journals are themselves of high quality. Conversely, many publications in lower quality journals are of high quality. Both outcomes can result in serious measurement error when ranking departments according to a journal quality index, especially for small departments due to non-cancelling errors”.
(This issue was also raised by Andrew Oswald in a paper in Economica, 2007: http://onlinelibrary.wiley.com/doi/10.1111/j.1468-0335.2006.00575.x/abstract. Oswald states that: “it is better to write the best article published in an issue of a medium-quality journal such as the OBES than all four of the worst four articles published in an issue of an elite journal like the AER. Decision-makers need to understand this”).
Surely citation indices would be the answer to the above problem, one might ask? Not the impact factor of the journal but the impact factor of the article published in the journal.
However, I think an ancillary point is that the RAE is an assessment of very recent research; and that there is a time-lag in articles building up citations (especially given the delays in the modern publishing process). Oswald (2007) uses data from the preceding 25 years and states that: “the best article in an issue of a good to medium-quality journal routinely goes on to have much more citations impact than a ‘poor’ article published in an issue of a more prestigious journal. This fact may not be known to all of the people who sit on funding councils, or perhaps even to many economists”.
Nonetheless, we know that contemporaneous evaluations of journal impact factors matter for hiring, for promotions and for certain funding competitions. And that there is some rationale for this; even if one just considers publishing in a certain journal of repute as a “signal” – a signal that the scholar might publish some highly cited (potentially ground-breaking or field-changing) research at some point.
Finally, one might wonder about the the extent to which the outcomes of the 2008 Research Assessment Exercise in the UK (determined by peer review) can be explained by a set of quantitative indicators. Jim Taylor (2011) addresses precisely that question: http://onlinelibrary.wiley.com/doi/10.1111/j.1467-8551.2010.00722.x/abstract
Taylor reports the following: “Each of the three components of research activity (namely, research output, esteem and research environment) is highly correlated with various quantitative indicators. A further finding is that the judgement of the Research Assessment Exercise panels was biased in favour of Russell Group universities. There is also evidence of bias by the economics and econometrics panel. The results support the use of quantitative indicators in the research assessment process, particularly a journal quality index.”
For those who are interested, the RAE (research assessment exercise) is set to be replaced with the REF (research excellence framework) in 2014: http://en.wikipedia.org/wiki/Research_Excellence_Framework
One major difference in the approach to be taken by the REF is described here (http://www.independent.co.uk/news/education/higher/andrew-oswald-ref-should-stay-out-of-the-game-1827306.html) by Andrew Oswald:
“Its central innovation is that it wishes to reward impact outside academia, and the plan is to put 25 per cent weight (in other words to allocate a quarter of UK university research-funding) according to this impact.”
Oswald is highly critical of the REF; his criticism of the reward for impact outside academia can be summed up with this quote from his article:
“It will get universities to put a lot of effort into doing things that they are not meant to do.”
Pingback: On the QS university rankings | Kevin Denny: Economics more-or-less
Pingback: Do college rankings matter for student choice? | Kevin Denny: Economics more-or-less
Pingback: The Irish Economy » Blog Archive » Do college rankings matter for student choice?
All comments are duly noted.
Look at the simplistics.
Universities were once the hallowed ground of those people who thought differently to the main stream person and who were able to validate their argument or were prepared to listen to someone elses argument in a reflective and constructive manner.
You all know what the prologue is, the “University” debacle that we have now.
Russ