Improving performance support systems through information retrieval evaluation.

This study examines existent and new methods for evaluating the success of information retrieval systems. The theory underlying current methods is not robust enough to allow testing retrieval using different meta-tagging schemas. Traditional measures rely on judgments of whether a document is relevant to a particular question. A good system returns all the relevant documents and no extraneous documents. There is a rich literature questioning the efficacy of relevance judgments. Such questions as, Relevant to whom? When? and To what purpose? are not well-answered in traditional theory. In this study, two new measures (Spink's Information Need and Cooper's Utility) are used in evaluating two search tools (tag-based and text-based), comparing these new measures with traditional measures and each other. The open-source Swish text-based search engine and a self-constructed tag-based search tool were used. Thirty-four educators searched for information using both search engines and evaluated the information retrieved by each. Construct measures, derived by multiplying each of the three measures (traditional, information need, and utility) by a rating of satisfaction were compared using two way analysis of variance. This study specifically analyzes small information systems. The design concepts would be untenable for large systems. Results indicated that there was a significant correlation between the three measures, indicating that the new measures provide an equivalent method of evaluating systems and have some significant advantages, which include not requiring relevance judgments and the ability to use the measures in situ.

Human Performance Technology and Performance Support Systems

A robust sub domain of study within instructional systems is human performance technology (HPT). HPT focuses on terminal performance as its unit of measure and explores three areas to seek leverage in improving performance--information, instrumentation, and motivation (Gilbert, 1996). To engineer performance, one strives to use strategies that have the greatest impact for the least cost. Accordingly, one may have a greater impact on improved performance, for example, by providing better information during the course of performance rather than designing a training intervention. Rossett (1996) distinguishes between training interventions and performance support by identifying the goal of training as building capacity, occurring before a need arises. Performance interventions, in contrast, provide support at the point of need, both in time and in place.

Information provided just in time offers a powerful tool for improving outcomes. Computer-based systems that can provide just-in-time information have been investigated since the 1980s. This class of interventions has variously been called electronic performance support systems (EPSS), performance support systems (PSS), and performance support tools (of which a performance portal is a subset). EPSSs are part online help, part online tutorial, part database, part application program, and part expert systems. EPSSs quickly and easily provide answers to the questions workers have when performing a job, and address workers' concerns (Carliner, 2002).

While there is a great diversity of functionalities across EPSS applications, one of the most common functions is a database of information that may be searched. In more embedded systems, this search is less obvious, often completed by the system based on context. However, in the more extrinsic systems, the search/retrieval function is an obvious and essential part of the system. "The primary design goal of an EPSS is that the knowledge it contains be easily retrievable by the users at the time they need it" (Cole, Fischer, & Saltzman, 1997, p. 50).

However, while the search/retrieval function or system has been shown to be an important function of performance systems, there is little known about within the field of instructional technology about evaluation of information retrieval systems. This is a problem, as there have been ongoing attempts to improve search systems. In 1992, Carr wrote, "Despite years of work, our methods for retrieving information from data bases remain relatively rigid and primitive" (1992). Much of the work concerning meta tags (Dublin core, IMS, SCORM, GEM, etc.) is predicated on the idea that adding meta tags can improve retrieval systems. Unfortunately, without being able to measure the effect of different search/retrieval systems, there is no way to know if efforts to improve practice are yielding results. The goal of this research is to investigate how the constructs and measures of information retrieval evaluation may inform the research agenda of performance support and if new constructs and measures may enrich and expand the theory.

Evaluation of information retrieval systems (whose practitioners are usually located in schools of library and information science) is a relatively mature field, with a literature dating to the mid 1960's (further if one considers pre-computer methods). While this literature offers a rich foundation, the constructs and measures underlying the theory are thin. For one wishing to evaluate information retrieval within performance support systems, additional constructs and measures that reflect the unique needs of performance support are necessary.