Unobtrusive Data Analysis of Digital Reference Questions and Service at the Internet Public Library: An Exploratory Study.

ABSTRACT

THIS ARTICLE PRESENTS THE RESULTS OF AN exploratory study attempting to establish a methodology for the unobtrusive analysis of a major digital reference enterprise. Logs of over 3,000 questions asked of the Internet Public Library in early 1999 were analyzed on the basis of questions asked (subject area, means of submission, self-selected demographic information), how those questions were handled (professional determination of subject and question nature, questions sent back to users for clarification), and answered (including time to answer) or rejected. In addition, answers that received unsolicited thanks were analyzed separately. Users seem to have difficulty in assigning subject categories to their questions, and to determine whether they are factual or require sources for assistance, and these decisions were often overridden by question administrators. The median time to answer questions was just over two days, and about one in five answers received thank-you messages from users.

INTRODUCTION

The advent of digital reference creates for librarians many new opportunities. Most of these revolve around new ways of offering service--i.e., at different times, to different populations, via different media, etc. However, since reference services delivered through the Internet are mediated in a chiefly textual environment, digital reference services also afford us new ways of examining the activities of reference. At the Internet Public Library (IPL), we have been providing digital reference services to our international patron group since opening on March 17, 1995, over five years ago. During that time, we have kept an exact record of every reference interaction that we have handled, over 40,000 questions to date. In this article, our goal is to explore just what sort of things can possibly be learned by examining this record.

As this is an exploratory study, we have limited our data set of interest to the questions received during the three-month period from January to March 1999. This period provides over 3,000 questions to examine. We are also purposely limiting inquiries to rather elementary data analysis--no content analysis or direct patron inquiries--as we are primarily interested in what sort of data can be drawn out of the amalgamation of questions via automatic means. In short, we want to know if anything useful can be learned about a digital reference service without investing a huge amount of resources.

As is typical with these sorts of studies, our explorations raise as many, if not more, questions than answers. In the conclusion of this article, we examine the more complex inquiries that are suggested by our elementary data analysis. We also consider ways in which the service itself might be modified to allow for more and more complex information to be gathered non-intrusively.

Our research questions were:

* What are important characteristics of questions and users (user-assigned subjects, self-identification of users)?

* How frequently do IPL administrators override user-defined subjects and nature of questions?

* How frequently do IPL question-answerers use internal features of the question-answering system?

* How long do answerers take to answer questions?

* Who sends thank-you messages back to the IPL?

* What are important characteristics of rejected questions?

REVIEW OF RELATED LITERATURE

Although digital reference services have been a part of libraries for some time, most of the literature has been anecdotal in nature. The few studies that have been done have generally focused on the nature and existence of these services (e.g., Janes, Carter, & Memmott, 1999, for academic libraries; Garnsey & Powell, 2000, for public libraries) and not any sort of qualitative or quantitative approach to the results or outcomes of these services. In a sense, this study is in the tradition of the numerous studies involving the evaluation of traditional reference services (e.g., Hernon & McClure, 1987; Durrance, 1989) as well as transaction log analysis (Peters, Kaske, & Kurth, 1993). The unobtrusive nature of our study shares some of the inherent limitations of transaction log analysis; as Kurth (1993) states: "Transaction log data ... don't reflect, except through inference, who enters the searches, why they enter them, and how satisfied they are with their results" (p. 98). However, we are unaware of any previous studies of digital reference services and, as such, are taking the first small steps into a new area of inquiry with this study.

OVERVIEW OF INTERNET PUBLIC LIBRARY REFERENCE

Internet Public Library reference has been covered in detail in many other places (e.g., Lagace, 1999; Lagace & McClennen, 1998). However, we feel that it would be instructive to provide first a brief overview of the process before diving into the data.

Users are invited to ask their questions by completing one of two forms: either a general purpose form …