Online and traditional paper-and-pencil survey administration: examining experimenter presence, sensitive material and long surveys.

Abstract: The present study examined response differences between paper-and-pencil versus computer administered surveys as a function of experimenter presence, survey length, and sensitive material, including sexuality-related measures. Three conditions were employed, one traditional supervised paper-and-pencil condition, one online computer administration condition that was supervised, and one unsupervised online computer administration condition. Missing data were greatest in the paper-and-pencil condition specifically at the end of this long survey in comparison to both online computer conditions. Of the four measures assessing sensitive material, more missing data were found in the paper-and-pencil condition but only for one measure assessing sensitive content. Researcher presence did not have an impact on performance. These findings have important implications for designing and interpreting survey outcomes.

Key words: Survey administration, sensitive measures, length of survey, computerized surveys.

Acknowledgements: We would like to thank Meghan Bellamy and Jen Caron for their help in conducting this research.

Introduction

For decades, paper-and-pencil surveys have served as the primary means for gathering survey information primarily because paper-and-pencil surveys provided a more economical and efficient means for collecting large amounts of data relative to other available data collection techniques, such as interviewing. Advances in computer technology, particularly, the prevalence and accessibility afforded by the Internet, have made it possible to reach vast numbers of participants even more efficiently and economically than through paper-and- pencil surveys (Pasveer & Ellard, 1998). In addition, software technology permits direct data entry thereby reducing costs as well as decreasing data entry errors and the time from data collection to analysis. With the transition to new and different mediums for presenting surveys, researchers and test-developers have questioned whether the new medium will affect the integrity of the survey instrument both in terms of reliability and validity issues and in response rates. The perceived advantages of computer and Internet versus paper-and-pencil survey administration have been supported in some recent investigations (Knapp & Kirk, 2003; Pettit, 2002; Truell, Bartlett, & Alexander, 2002) while others have identified limitations (Crawford, Couper & Lamias, 2001; Tourangeau, 2004).

Several studies assessing the reliability and effectiveness of computer versus and paper-and-pencil survey methodologies have found none or relatively few differences between the two methodologies (Chuah, Drasgow, & Roberts, 2006; Cronk & West, 2002; Knapp & Kirk, 2003; Pettit, 2002; Truell et al., 2002). Given these findings one might conclude that the two methodologies yield equivalent products, with online presentations offering the additional economic advantages. There are however, differences in the methodologies that may differentially affect outcomes.

While most Internet and some computer assessments eliminate the need for a researcher to be present during administration, the anonymity and lack of proctoring may affect how participants respond to survey items. Without having a researcher present to guide participants, there is the possibility of questions being misunderstood. With no one present to clarify the question, confusing items may be answered randomly or abandoned (Nosek, Banaji & Greenwald, 2002). Legitimacy of the survey may also be compromised in the absence of a researcher. For example, the survey may not be taken as seriously by the participant (Tourangeau, 2004) and may be completed by someone other than the desired participant.