The critical incident technique and the qualitative evaluation of the connecting libraries and schools project.

ABSTRACT

This article describes Flanagan's Critical Incident Technique (CIT) for those seeking guidance in its application. Examples in the library and information science (LIS) field are discussed, including an indepth example of a CIT study conducted as part of a qualitative evaluation of the Connecting Libraries and Schools Project (CLASP) in New York City. The CLASP study analyzed critical incidents from 2,416 fifth and seventh grade students regarding their perceptions of interactions with urban public librarians and library staff. For both positive and negative critical incidents, the most important factor in these preadolescent's perception of successful library visits is the attitude of the librarian or staff member they encounter.

INTRODUCTION

This article discusses Flanagan's (1954) Critical Incident Technique (CIT) and describes its applications in the library and information science (LIS) field. The CIT is a qualitative method designed to draw out the most memorable aspects of an event or experience from the study's participants (Ruben 1993). It has been used to evaluate programs or services and to inform their improvement. CIT questions typically have this format: Remember a time when you had a successful (specify activity)? Please describe. What was it about (specified activity) that made it successful? Or the negative: Remember a time when you had an unsuccessful (specify activity)?

Guidance is provided by this article for those considering using CIT in their research by discussing an example of a CIT study conducted by the author as part of a qualitative evaluation of the Connecting Libraries and Schools Project (CLASP). CLASP, a citywide program of The New York Public Library, the Brooklyn Public Library, and the Queens Borough Public Library, was evaluated by collecting critical incidents from questionnaires completed by 2,416 fifth and seventh grade students to gather their perceptions of encounters with public librarians.

USING THE CRITICAL INCIDENT TECHNIQUE IN A LIBRARY SETTING

CLASP was created by The New York Public Library in 1991 through a grant from the DeWitt Wallace-Reader's Digest Fund and was expanded to all five Manhattan boroughs in 1994 (Tice, 2001). The author was asked to conduct a qualitative evaluation of the impact of CLASP on students and chose the CIT as the most suitable method. Gathering students' critical incidents allowed analysis to center on how effective CLASP had been in changing or improving student's attitudes toward the public library. (1) The study had a twofold purpose: (1) evaluating CLASP, and (2) advancing knowledge of preadolescent perceptions of librarian/library staff encounters in the urban public library setting. (2) The CIT was selected as an appropriate method for achieving these purposes.

Flanagan's (1954) seminal article on the CIT documents the technique's origins and provides guidelines for its use. According to Flanagan, the CIT has five key stages, each of which will be discussed below with examples of its application in the CLASP evaluation:

1. General aims

2. Plans and specifications

3. Collecting the data

4. Analyzing the data

5. Interpreting and reporting

Stage 1: General Aims

During this stage, the key decision is to determine the purpose of the study: What does the researcher want to find out (Redman, Lambrecht, & Stitt-Gohdes, 2000)? Also, the researcher needs to conduct a literature review to discover what is already known about the type of activity, program, or participants to be studied. The CIT is useful for evaluating particular activities such as conducting outcomes assessments (Bycio & Allen, 2004; Jacoby & O'Brien, 2005) and measuring customer satisfaction (Arnold, Reynolds, & Ponder, 2005). Specifically, the researcher using CIT is looking to find out "precisely what it is necessary to do and not to do if the activity is to be judged successful or effective" (Fisher & Oulton, 1999, p. 113). In the case of the CLASP project, the study's purpose was program evaluation.

Stage 2: Plans and Specifications

Before any data can be collected, sufficient preparation is necessary. One important decision is identifying the subjects from whom critical incidents will be collected. In the case of the CLASP study, fifth and seventh graders, aged nine to twelve years old, were targeted because they would have experienced several CLASP classroom visits and they would have the writing skills necessary to complete the questionnaires. This highlights another decision to be made--the method of data collection. Critical incidents can be collected by observation, face-to-face (individual or group) interviews, telephone interviews, or paper or email questionnaires. This decision will be influenced by budgetary and staff considerations. In the CLASP study, project librarians were available to administer the paper questionnaires at the various school sites, to distribute and collect the surveys, and to assist students as needed.

Data collection through observation or face-to-face, group, or telephone interviews is much more labor-intensive when compared to paper or email survey distribution, but it has the advantage of providing the ability to ask probing or clarification questions of the subjects. This leads into the next decision point--determining who will collect the data. For a local project, such as the CLASP review, existing staff can be used. It is possible for a single researcher to collect critical incidents, even for multiple research sites (see Ozkaramanli, 2005; Radford, 1993, 1996, 1999). For a project with a larger, perhaps national, scope, it is necessary to gather a team of researchers (Redman, Lambrecht, & Stitt-Gohdes, 2000).

It is important to develop a plan for recruiting subjects. In the public or academic library setting it may be necessary to over-recruit and to offer an incentive such as the $15.00 value card issued to student interview participants reported by Jacoby and O'Brien (2005). Even with this incentive, only five of twelve students recruited actually showed up for the interview (Jacoby & O'Brien, 2005). Group interviewees are frequently recruited with offers of refreshments. Budget considerations usually dictate the range of incentives that can be offered.

Stage 3: Collecting the Data

For all data collection methods, a data collection instrument (survey or interview schedule) must be constructed and training materials or instructions for the data collection team must be developed. It is critically important that those collecting the data be thoroughly trained. Redman, Lambrecht, and Stitt-Gohdes discuss specifics of conducting the CIT interview, which they recommend as a "powerful tool" for data collection (2000, p. 136). For group or individual interviews, it is highly recommend that they be audio-taped with permission of the participants (see Radford, 1993, 1999).

In determining what questions to ask, it is useful to visualize the final report or paper to be written and to think about what questions will help to gather the information needed for that report. In commercial marketing, CIT questions could be as simple as "What did you like most (least) about your airline flight today? Why?" For the CLASP project, each student was asked to recall and describe in their own words: (a) a successful library experience either recently or in the past (that is, a positive critical incident), (b) an unsuccessful library experience (that is, a negative critical incident), and (c) the factors that made the experience successful or unsuccessful. Flanagan (1954) reported that…