Ranking journals in the humanities: an Australian case study.

Librarians have been engaged in the evaluation of journals for almost as long as journals have been published. The methods which we have used for evaluation have often been rather homespun--sometimes little more sophisticated than 'I know my users and I know what they want'. However in recent years, with dramatically rising serial prices, the evaluation of the journal collection has become more of a challenge for academic librarians, and we have had to look for more intellectually rigorous methods.

Somewhat to our surprise, we have discovered that journal evaluation is an issue which is now of interest to many of our clients as well. As funding bodies put pressure on universities to demonstrate the value of their work, the question of how to measure the quality of research outputs has become critical for universities and for those who work in them. The Australian Government, for example, has recently established the Research Quality Framework 'to develop the basis for a more consistent and comprehensive approach to assessing the quality and impact of publicly funded research.' (1)

Any such assessment must be influenced by a variety of factors, but inevitably outputs in the form of scholarly publications will play a significant role, and in many disciplines the journal article is the standard form of publication. As Butler has demonstrated, pressure from funding bodies has led to an increase in journal publication by Australian academics during the last decade, but much of that increase has occurred in journals of lower quality. (2) This has inevitably prompted a closer scrutiny of the quality of the journals in which academics are publishing. The report on consultations held in Australia early in 2005 concerning the Research Quality Framework records that:

    The most contentious area of discussion was around  whether  
   rankings of journals could be provided for different disciplines  
   that would allow review panels to rate publication performance  
  without having to actually read publications. Some (primarily in  
   the humanities and social sciences) suggested that this was    not  possible and that panels must actually assess papers put    forward.  Others (primarily in the sciences) maintained that it was    unrealistic  to expect panels to review papers and that it should    be possible to  reach broadly agreed lists of journals for each    research field. (3)  

Methods of Journal Evaluation in the Humanities

As the passage quoted immediately above makes clear, journal ranking is a particularly thorny issue in the humanities. At the most fundamental level, some academics would argue that evaluation of journal output is an unprofitable exercise, as the most important research outputs in the humanities are published in books. For example, a recent report from the (Australian) Council for the Humanities, Arts and Social Sciences states that 'standard bibliometric practices do not capture the variety of research outputs (for instance books, documentaries, policy reports) in the humanities and social sciences, and are clearly absurd in the creative arts' (4) and a recent Canadian review of bibliometric practice in the social sciences and humanities concluded that 'the research article clearly does not play as central a role in the SSH [social sciences and humanities] as it does in the NSE [natural sciences and engineering]. (5)

Let us put aside these objections and accept, for argument's sake, that journal articles are a significant output medium in the humanities, even if they are not necessarily the dominant medium. Are there methods which can be used to evaluate and rank journals in the humanities? And how applicable are those methods to Australian journals?

Citation Counts and Impact Factors

The citation indexes developed by the Institute for Scientific Information (hereafter ISI) have established themselves as the major tool for bibliometric evaluation of scholarly journals. The impact factors which ISI calculates for journals in the sciences and social sciences are widely quoted as measures of the quality of those journals and, by extension, of the articles which they publish.

However, in the arts and humanities no such impact factors are calculated. The method used for calculating impact factors in the sciences and social sciences is based on a three-year cycle of publication and citation. As ISI have found,

  citations in the Arts and Humanities ... do not necessarily follow  this same predictable pattern as citations to Social Sciences and  Natural Sciences articles. Citations to an article on the 19th  Century Romantic novel, for example, may accrue slowly at first,    and  then slacken, fluctuating over time in cycles consistent with  scholars' varying interest in the topic. (6) 

Thus, for want of an agreed methodology, no impact factors are available in the humanities, although it is certainly possible to use the data in the Arts & Humanities Citation Index to count citations to humanities journals. The problem lies in deciding how to analyse such data so as to provide meaningful comparisons between journals.

There are other problems in using the ISI indexes for journal evaluation. Much humanities research is published in books, but the ISI indexes only record citations in journals, so citations in books to the journal literature are not represented in the indexes. The 1,100 source journals scanned for the Arts & Humanities Citation Index are only a fraction of the scholarly journals currently published, and they are predominantly English-language journals with an international focus. As the founder of ISI has stated, 'unless a journal of interest to only a small region of the world is exceptional in some way, we are less likely to cover it' and 'we do cover a large number of foreign-language journals, but the presence of informative abstracts or summaries in English is essential.' (7)

Journals dealing with topics such as Australian literature or Australian history are most likely to be cited in other Australian journals. Only 13 Australian journals in the humanities are currently ISI source journals (see Table 1). The…