Search patterns of remote users: an analysis of OPAC transaction logs. (online public access catalog)

The focus of this study is the search behavior of remote users of the University of California MELVYL Library System, an online public access catalog (OPAC). Transaction logs from randomly selected remote user search sessions are analyzed. Descriptive data on the number and type of searches, choice of search mode and database, number of retrievals, number and type of errors, and use of system HELP facilities are presented. The search data have been cross-tabulated with demographic data on the same group of remote users, collected through an online survey conducted by the authors. Effectiveness of system usage is discussed. A case is made for the desirability of additional heuristics in the catalog portion of the system.

The MELVYL Library System of the University of California (UC) first became accessible outside the library setting in the mid-1980s. Remote usage has risen steadily since that time and typically accounts for more than one-third of the half-million queries entered in the system each week during peak usage periods.[1] In an effort to understand more fully this growing user population, the investigators undertook a two-part study. The findings of the first part of the study, an online survey of users who accessed the MELVYL system from outside the library setting, were reported previously.[2] The present report contains the results of the second phase of the study. In this phase, the investigators coded selected data from the transaction logs of the surveyed group, used microcomputer programs to compare those data by user status and other user characteristics, then visually reexamined the user command portions of many of the individual logs to gain further insight into user search behavior.

THE MELVYL LIBRARY SYSTEM

The MELVYL system provides access to nearly eight million monograph and periodical titles held principally by libraries of the University of California.[3] In addition, the system offers its users access to several periodical index databases and serves as a gateway to many other specialized databases and library catalogs. Users may access this rich array of resources directly from their homes, offices, or other sites, through dial-up or networked connections.

The MELVYL system began as a prototype online catalog for the University of California, a nine-campus, doctorate-granting institution, which currently supports a main library on each campus, nearly one hundred branch and specialized libraries across the system, and an enrollment of more than 166,000 students. The system serves as a union catalog, to which the campus cataloging agencies contribute their records. Most of the campus libraries implemented local online catalogs during the 1980s; these serve as their primary catalogs and as a gateway to the UC union catalog residing within the MELVYL system.

After a decade of development, the catalog portion of the MELVYL system has achieved the status of a second-generation OPAC. It generally reflects Charles Hildreth's hypothetical construct of features that constitute "a qualitative leap of progress over first-generation online catalogs."[4] For example, the system supports keyword access to a variety of fields, explicit Boolean search logic, limiting capabilities, optional and automatic truncation of search terms in some kinds of queries, extensive help facilities (including contextual help screens), and multiple display formats. Some examples of special processing introduced to improve retrieval are the "normalization" of search terms in several fields and the treatment of title words as "exact" titles under certain conditions.[5] Appendix A contains a summary description of system commands and indexes.[6]

Hildreth states that researchers involved in information retrieval generally acknowledge that "today's conventional keyword-indexed, inverted file, Boolean logic search and retrieval systems like BRS, DIALOG.... LEXIS-NEXIS (and all second-generation OPACs) are powerful and efficient but are dumb, passive systems which require resourceful, active, intelligent human searchers to produce acceptable results."[7] Indeed, retrieval systems like the MELVYL system are the focus of continued study designed to solve the problems they still pose to the user.[8]

Clifford Lynch, director of the UC Division of Library Automation, has predicted that "the evolutionary descendants of the MELVYL system (and other systems of its generation) will differ in many ways from today's online catalogs.... Not only will the user interface and searching algorithms change, but the contents and scope of the information bases to which the system provides access will change also. These future systems will be more heuristic, and will evaluate information and guide the naive user, while still permitting the |expert' user total and direct control."[9]

The advent of next-generation OPAC development is an exciting prospect, but it is not yet clear how some of the major design challenges will be addressed. Hildreth does not expect a "giant, discontinuous leap forward to the next generation of online catalogs" but, rather, that "progress is likely to be made in small, incremental steps."[10] Lynch and his associates suggest that the evolution of the existing MELVYL system interface and other elements of its functionality will continue to evolve over the next decade.

Because the investigators had a unique opportunity to include demographic data in their analysis of transaction logs, the principal aim of the present study has been to compare the remote use of the MELVYL system by various characteristics of remote users. The factors expected to make the most difference were status and frequency of use of the system. The data support that hypothesis, as reported below in the Analysis section under subject searching and in the Effectiveness of Searches section, where those factors were particularly apparent. Also investigated were the types of problems remote users encountered in trying to make effective use of the system and the degree to which the system helped or failed to help users overcome those problems.

COMMENTS ON TRANSACTION LOG ANALYSIS

Recently published reports of OPAC transaction log studies by Rhonda Hunter, Steven Zink, Thomas Peters, and Sally Kalin were examined for methodological relevance to the present study.[11] All of these interesting studies, which analyzed actual user OPAC searches, were conducted by public service librarians in academic libraries. Their methodologies varied considerably, but their aims were essentially the same: to collect data relevant to instruction in catalog use or to determine whether changes in on-screen user instructions might improve user success with their respective OPACs. In most of the studies, sampling was employed, printouts of logged data were visually examined and coded, and tabulations of selected data were reported Only Kalin's study focused directly on remote users.[12] None of the studies incorporated demographic data about the users.

A decade ago, John Tolle noted the difficulty in comparing OPACs through analysis of transaction logs because the logging systems vary as to the information they provide.[13] Even if formats for transaction files were standardized, a suggestion made several ago by Lois Ann Colaianni,[14] the considerable design variations among OPACs would continue to pose difficulties to investigators trying to compare user performance across systems. The present study does not escape this problem of comparability, but special effort has been made to focus on data that would tend to be of interest beyond the system being studied.

A principal goal has been to provide basic demographic and search behavior data about patrons who use an OPAC outside the academic library setting and who, therefore, must typically depend on the system itself to solve the problems they encounter. As reported previously, just over half (50.9%) of remote users rarely or never use the MELVYL system at public terminals in a UC library. Even among UC-affiliated remote users, that figure is about 33%. Interestingly, 32.8% of the remote users are either first-time, rare, or monthly users of the system.[15] Since at least one-third of all searches in the system are now conducted by remote users (a phenomenon only a few years in the making), the experiences of this population merit close attention.

Transaction logging data have been collected by the developers of the MELVYL system since its inception in order to monitor system performance, provide information on system usage, and support special analyses like the present study.[16] Statistics extracted from the logging data tell a great deal about system usage but do not prepare one for the sometimes startling observations gained through reading the user command portions of the transaction logs.

Several problems arise in the design and execution of studies of the type reported here. For example, when logs of user commands are to be examined by investigators, the scale of the study is a leading determinant in the design. The study may be confined to a brief, representative usage period, to particular types of transactions, to particular user populations, or to a combination of any of these or other elements. Methodologies that include the review of the actual displays resulting from a user's search typically require the investigator to replicate the search shortly after the logging data are captured.

Some problems arise in the course of visual analysis and interpretation. A user's intent may be clearly reflected in some logs but elusive in others. In the …