Updated: 4 June 2008
The preliminary version of a paper that profiles departments and ranks the top scholars working in Ireland on politics, government, and international relations according to scholarly impact (measured by citations) has now been posted for comments at http://www.tcd.ie/Political_Science/staff/kenneth_benoit/citations/.
The current version of the paper is dated April 18, 2008.
Here is a Dilbert cartoon that offering one apparently common view that may apply to bibliometrics:
There has also been some interesting correspondence regarding our paper. On April 24, 2008 we received this official letter from Professor Jennifer Todd, Chair of the Research and Innovation Committee from UCD’s School of Politics and International Relations. (This letter was also publicly sent to the Political Studies Association of Ireland mailing list.) Our reply was also posted to the PSAI mailing list:
Dear Jennifer, Thank you for your letter dated 25 April 2008 about the data collection stage of our planned paper, “A Relative Impact Assessment of the Academic Study of Politics In Irelandâ€. As we originally stated quite clearly in our draft paper and on our website, we had posted the data we collected not because we intended for this data to form our final analysis, but rather in the interests of full accountability to those involved, especially for the purpose of catching errors from those most familiar with their own citation data. Having served this purpose, the tables you refer to were removed as planned on April 30, 2008. As for your offer of assistance, we would be delighted were the PSAI to appoint a subcommittee to assist with additional data collection for our paper. While our present concern is with assessing research impact in early-to-mid 2008, furthermore, we also welcome your suggestion that the bibliometric information we have been gathered be updated on a regular basis. We are ready to cooperate immediately on such a venture, but just in case the PSAI machinery moves too slowly, we have also undertaken independently to gather the additional data you and the SPIRe Research and Innovation Committee suggested would be vital to our analysis. Here we refer to the citations from ISI’s “Cited Reference Search.†Adding this additional data will mean that our exercise examines four measures in total: ISI’s article citations, ISI’s more inclusive (but less quality-controlled) “cited reference search,†Scopus’s article citations, and Google Scholar’s results which tend to include just about everything. If you can think of any more empirically observable, third-party measures of scholarly impact by which scholarly impact could also be compared for politics scholars in Ireland, we would be glad to consider including those as well.The response attached to your letter from the SPIRe Research & Innovation Committee raises a number of other interesting points on which reasonable people might simply disagree, and we think it unlikely that any subcommittee or other formal collective forum will be sufficient to resolve this disagreement. In particular we think it unlikely that any objective, comparative assessment of scholarship can avoid the purported “bias [from] a culture of extensive mutual citation, rather than reliance on original data and sources.†The focus of our analysis is on citations of published scholarly work, and this is unlikely to change. Of course, science and scholarship more generally tend to make advances in a cumulative, often adversarial fashion. Just as we are free to compile data from publicly available sources such as ISI, Scopus, and Google Scholar, you and your colleagues are free to offer rejoinders to our paper or present alternative comparative assessments of scholarly impact. Indeed, we look forward to the healthy debate focused on research impact that would result from such an exchange.
Sincerely yours, Kenneth BenoitMichael Marsh
Department of Political Science, Trinity College
On June 4, President of the PSAI (Political Studies Association of Ireland) sent the following e-mail to all members of the PSAI mailing list:
The Heads of Departments of politics in Ireland met in Dublin recently to discuss matters of common interest to the profession. These included CSPE in the leaving cert and the impact of Bologna on politics degrees. The most contentious issue, however, was that of research measurement highlighted by recent research by colleagues in TCD. It was recognised that politics departments compete less with each other than with other disciplines in their internal university battles for resources. Metrics may be an inevitable by product of changing funding structures. Nevertheless, the Heads felt it would be useful if the PSAI could facilitate a consensus on the range of appropriate measures. It was agreed, therefore, that the Association would convene a meeting of nominees from each department to share views on what metrics might be appropriate for use in combination or singularly within each institution. The PSAI may also invite members with a particular expertise in the issue of metrics. The proposed date of the meeting is 18th June.Neil Collins
