Meta-analysis in library and information science: method, history, and recommendations for reporting research.

ABSTRACT

Meta-analysis is a method for summarizing statistical findings across multiple research studies. It is a useful method for assessing the level of agreement or disagreement surrounding a given research question. The ability to perform meta-analysis is dependent on the level of consistency in measures and the amount of data shared in published research. Guidelines to minimum standards for reporting research may improve the quality of writing in published research. Inconsistencies in reporting research findings across studies, failing to provide enough detail on method and instrumentation to facilitate replication, and the multiplicity of different operational definitions or measures for the same concept all pose difficulties to successfully attempting any form of research synthesis. This article presents a methodological explanation of meta-analysis, a literature review describing the application of meta-analysis in library and information science, and guidelines for reporting quantitative research that would enable subsequent researchers to perform meta-analysis.

INTRODUCTION

Every scholarly journal provides highly precise guidelines to its authors regarding the length of articles, the formatting of manuscripts, and the style of citations and footnotes. While authors may meet these guidelines with varying degrees of success, at least all parties involved in the scientific communication process recognize that a standard has been established. Curiously, few scholarly journals provide any guidelines regarding standards for the reporting of research in terms of the descriptive elements of a dataset that should be shared, the statistics that should be presented for a given method of analysis, and whether or not a copy of the instrument should be included. One reason for this omission in the field of library and information science (LIS) may be because of the variety of disciplinary and methodological approaches being used by researchers. To impose rules for the reporting of research might curtail the creative freedom of authors in presenting their work. However, this rich variety of quantitative and qualitative methods and different disciplinary orientations argues all the more for such guidelines to be established. For example, whereas physics or economics may have more rigid rules for publishing research that are well understood by researchers in their respective disciplines, LIS encompasses a much broader array of research methods that is harder to explicitly articulate. How does a researcher specializing in information retrieval working with a database of 10,000 records and hundreds of queries know how to evaluate a piece of research on information behavior based on twenty in-depth interviews? How does a researcher studying information services who reviews thousands of virtual reference transactions understand the validity of a philosophical investigation in classification theory? Such confusion may grow worse when LIS researchers examine the work of their colleagues in computer science, management, law, health informatics, or technical communications whose research questions may be similar to our own.

A guide to the minimum standards for reporting research may serve to help nonspecialists (as well as students) better understand what to expect when reading about a study employing a method with which they are unfamiliar. A second and perhaps more important benefit might be to improve the quality of writing in published research. Does the article provide enough detail so that the study could be replicated? Does the article then provide enough data so that results from a subsequent study could be compared to findings from the original study? Without replication, research in LIS advances haltingly, and validation of findings is difficult to achieve. The development of commonly accepted definitions and indicators for important concepts proceeds slowly. How do we measure information anxiety, collection strength, or user satisfaction? With the absence of a predominant method of observation, researchers often develop their own operational definitions for each new study. Even when discussing relatively concrete concepts such as number of volumes in the collection, different sources use different measures (compare the Association of Research Libraries [ARL] statistics to guidelines on counting given by various state libraries), and members of the ARL debate what it means to "own" volumes placed in a regional repository (ARL Committee on Statistics, 1997).

Inconsistencies in reporting research findings across studies, failing to provide enough detail on method and instrumentation to facilitate replication, and the multiplicity of different operational definitions or measures for the same concept all pose difficulties to successfully attempting any form of meta-analysis. Meta-analysis is a form of research synthesis, and the terms are used interchangeably in fields that rely heavily on quantitative methods. Meta-analysis is a body of techniques that enables researchers to draw conclusions based on the findings of previous studies and present them in a useful and compact fashion (Matt & Cook, 1994; Hunter & Schmidt, 1990). The benefit of meta-analysis is that it enables researchers to obtain a greater understanding of the nature of the association between outcome and independent variables by comparing different values of effect size gathered from a large body of research. The ability to summarize findings across multiple situations and discover consistent trends (or in some cases, inconsistent trends) is a critical component of scientific research.

The lack of common definitions and research…