The evidence base: where is it?(use of bibliometrics in evidence-based policy development)

In criminal justice, as in many other areas of public administration, politicians and managers increasingly call for evidence-based policy (EBP). This should raise questions about what evidence, who will find it and how will it be used. The rising emphasis on bibliometrics as the way to measure importance and impact is another development that should raise questions about what is being measured and how it will be used. The academic research publication model does not fit all aspects of criminology very well, where much work is done by or for government and non-government organisations, and much is multidisciplinary. This grey literature has always presented problems in meeting standards for peer-reviewed, evidence-based evaluation, but is often all there is--if it can be found. Funding agencies may have stringent requirements for evaluation studies, but how are these reports written or structured and what happens to them? How can they be used to inform subsequent practice?

Evidence and bibliometrics: simple solutions

In April 2008, the Australian Prime Minister (Rudd 2008) addressed his senior bureaucrats on his government's policy priorities for, and expectations of, the Australian Public Service. The third of his seven key points called for evidence-based policy. This is good news for the proponents of this approach like the Campbell Crime and Justice group, and indicates a commitment to good, research-based public policy decision making.

Also in 2008, the Australian and New Zealand Society of Criminology (ANZSOC) released its list of quality rankings for criminology journals and publishers (Brown & Daly 2008; Daly 2008) and Cohn and Farrington (2008) updated their work on influential criminologists. Although the particular Research Quality Framework assessment of Australian universities for which the ANZSOC ranking was done is no longer operative, it is expected that bibliometrics will play a part in whatever replaces it.

Ostensibly now, armed with these two developments, the approach to good policy is straightforward--at a minimum, the policy maker or practitioner can go to the best sources to find the evidence about what works. However, both raise the question of how this evidence base is created, found, shared and used.

The value of bibliometrics has long been debated (see Levi 1995), with the Cohn and Farrington article giving a good overview of its history in criminology and criminal justice. Its commercial genesis lies in the Institute for Scientific Information, established in 1960 with the Science citation index and now flourishing in the Web of Knowledge and other services. Although unlikely to go unchallenged anywhere, the value of citation analysis in science is likely to be less contested than in the social sciences.

Regardless of how impact analysis might be used to measure academic performance however, there is a disjunction between the literature as identified in such bibliometric exercises, the standards set in systematic analyses based on the medical template, and the information created and used in policy and +practice not so much for what is found, but for what is missed. The transition of graduates from university to policy and practice, with their experience limited to academic journal and commercially published monograph literature, particularly to online, instantly accessible literature, and the new emphasis on systematic reviews combine to mean that the complex wealth of experience may be masked.