Human research and data collection via the Internet.

Key Words anonymity, browser-based research, CGI, computers in psychology, computer programming, experimental dropouts, experimenter bias, HTML, HTML forms, Internet research, Java, JavaScript, methodology, missing data, on-line research, Perl, survey, Web experiment, WWW

* Abstract Advantages and disadvantages of Web and lab research are reviewed. Via the World Wide Web, one can efficiently recruit large, heterogeneous samples quickly, recruit specialized samples (people with rare characteristics), and standardize procedures, making studies easy to replicate. Alternative programming techniques (procedures for data collection) are compared, including client-side as opposed to server-side programming. Web studies have methodological problems; for example, higher rates of drop out and of repeated participation. Web studies must be thoroughly analyzed and tested before launching on-line. Many studies compared data obtained in Web versus lab. These two methods usually reach the same conclusions; however, there are significant differences between college students tested in the lab and people recruited and tested via the Internet. Reasons that Web researchers are enthusiastic about the potential of the new methods are discussed.

 
CONTENTS 
 
INTRODUCTION 
  Computer Developments that Set the Stage 
TECHNIQUES 
  Getting Started: HTML Forms 
  Client-Side Programming 
  Server-Side Programming 
  Run Your Own Server 
  Stimulus Delivery 
METHODOLOGY 
  Multiple Submissions 
  Dropouts in Between-Subjects Research 
  Recruiting Participants Via the Web 
  Sampling Bias and Stratified Analysis 
  Response Bias 
  Experiment or Bias 
PILOT TESTING IN THE LAB 
WEB VERSUS LAB 
SUMMARY AND CONCLUSIONS 

INTRODUCTION

In the last decade, a new protocol for sending information on the World Wide Web (WWW), hypertext transfer protocol (HTTP), created a new way to conduct psychological research. This new technique allows researchers to collect data from participants all over the world 24 hours a day and seven days per week. Surveys and experiments can be delivered quickly to anyone connected to the Web and data can be saved automatically in electronic form, reducing costs in lab space, dedicated equipment, paper, mailing costs, and labor. Once an experiment or survey is properly programmed, data can be stored in a form ready for analysis, saving costs of data coding and entry that used to be an expensive and time-consuming part of the research process.

Computer Developments that Set the Stage

By early 1995, a number of changes were under way that created the conditions for a new approach to psychological research:

* Computer hardware and software improved; it became less expensive to own computers and easier to operate them.

* The number of individuals who owned or had access to personal computers, email, and the WWW was increasing exponentially.

* More and more valuable content was being added to the WWW.

* Speed of connections to the Internet were improving and browsers, the programs that display WWW content, were now able to handle more complex media and programming languages of Java and JavaScript (initially called "LiveScript").

* Hypertext markup language (HTML) 2 supported the technique of forms.

The new standard of HTML 2, introduced in late 1994, supported forms, which allowed a person viewing a Web page to easily send data back to a designated server, which could process, code, filter, and save data electronically. This technique made it possible for a person, even without an email account, to be able to participate in a survey from an Internet-connected computer (e.g., in a public library or campus computer facility) even if that computer was not configured to send email.

Within a few months, psychologists began using this technique to collect data in surveys and experiments. A number of these "pioneers" contributed chapters to Birnbaum's (2000b) edited book, which included a chapter by Musch & Reips (2000) that summarized the (short) history of psychological experiments (not surveys) on the WWW [see also Musch (2000)].

Musch & Reips (2000) noted that the first psychological experiments (with manipulated variables) conducted via the Web were those of Welch & Krantz (1996) and Krantz et al. (1997). Krantz (1998) maintains a Website, "Psychological Research on the Net," (http://psych.hanover.edu/research/exponnet.html) that lists experiments currently running on the Internet. The site had 35 links to on-line studies on June 17, 1998. By May 11, 1999, there were 65 links, and on May 10, 2003, there were 150 links, including 45 to projects in social psychology and 30 to studies of cognitive psychology.

Several of these links lead to multiple experiments. For example, Jonathan Baron (http://www.psych.upenn.edu/~baron/qs.html) has been running about 50 studies per year from his site. The PsychExps site at Ole Miss (http://psychexps. olemiss.edu/) listed 33 lab and research studies on May 10, 2003, most of which were in cognitive psychology. Similarly, the student research project site at my Decision Research Center of Fullerton (http://psych.fullerton.edu/mbirnbaum/ decisions/thanks.htm) listed 21 studies on that same date. Ulf-Dietrich Reips's Web Experimental Psychology Lab (http://www.psychologie.unizh.ch/genpsy/Ulf/Lab/ WebExpPsyLab.html), which began operations in 1995, had links to 16 active studies in English and 12 in German plus about 80 in its archive on this same date in 2003. The Web Experiment List (http://genpsylab-wexlist.unizh.ch/) listed 166 studies in 2003. Although not all studies are listed in these sites (and some items are duplicates), the numbers give an indication of the rapid expansion of this method of doing research. A few hours visiting the links would convey to the reader a general impression of the kinds of studies being done this way.

Although any given work may have more than one focus, I divide my review in three distinct topic areas, according to my judgment of a work's main thrust. These three areas are (a) Techniques that describe of analyze "how-to" issues. For example, how can we randomly assign participants to between-subjects conditions? (b) Web methodology, which deals with internal and external validity of Internet research. For example, what are the threats to internal and external validity if people drop out of Web-based experiments, and what are the considerations of using particular techniques to reduce such dropouts? (c) Web versus lab comparisons, which includes findings specific to the Internet and comparisons of Web and lab studies of the same phenomena. For example, do experiments in decision making that are conducted via the Web yield the same conclusions as those done in the lab? In what ways do results obtained in Web and lab differ?

A fourth major area of research involves social psychology of the Internet, which deals with the Internet as a new social situation or communication medium. For example, do people exhibit different social behaviors on-line and in person? This topic is the focus of a separate review (Bargh & McKenna 2004) and will not be covered here except in relation to methodological issues in Web research.

TECHNIQUES

Many books and articles describe techniques that can be useful for conducting research via the WWW. Items that are primarily commercial and directed to the general audience (e.g., books on HTML, JavaScript, Java, Perl, etc.) are not summarized here, but those that are specifically directed to psychological researchers are discussed.

Getting Started: HTML Forms

Probably the easiest way to get started with the techniques of Web-based research is to make a survey or experiment using one of the free programs (e.g., Birnbaum 1998) to create the Web page for a simple survey of factorial experiment. These programs are available from the following URL: http://psych.fullerton.edu/ mbirnbaum/programs/.

SurveyWiz and FactorWiz are Web pages that make Web pages for collecting data via the WWW. Within each Web page is a list of instructions for use (see also Birnbaum 2000c, 2001a). Creating a Web form with SurveyWiz is as simple as typing the questions and pushing buttons for the type of input device desired. The program supports text boxes, which are boxes into which a participant can respond by typing a number or short answer. SurveyWiz and FactorWizRB also support radio buttons, which allow the user to click along a rating scale.

The most fundamental technique of Web-based research is HTML, the formatting language used to compose Web pages. A Web page (a document) can contain formatted text, links to other information on the Web, pictures, graphics, animations, and other media such as sounds or video. Many software products are available to create Web pages, including free ones such as Birnbaum's (2000c) FactorWiz, WEXTOR by Reips & Neuhaus (2002), Schmidt's (1997b) WWW Survey Assistant, and White & Hammer's (2000) Quiz-o-matic (which makes self-scoring quizzes). Despite the availability of free and commercial software to make Web pages, a Web researcher needs to have a basic understanding of the "tags" (commands) of HTML. There are many free tutorials available on the WWW for learning HTML, as well as many commercial books on the subject.

The only introductory text in English on Web research is Birnbaum's (2001a), intended for behavioral researchers who are new to Web research. It includes a brief introduction to HTML, including the technique of forms (Chapters 2-5), spreadsheet and statistical software useful in Web research (Chapters 6, 7, and 12-15), an introduction to JavaScript (Chapters 17-19), and a brief introduction to advanced topics such as Java, Authorware, and other methods (Chapter 20). Several chapters cover psychological content with examples of how to program the experiments, explanations of the examples, and instructions on how to analyze the data. A compact disk (CD) accompanies the book with many programming examples, experiments that are ready to run, and data for those experiments. There are also chapters on methodology, recruiting, and ethics of Web studies. Materials for using this book with both undergraduate and graduate students are available from the following URL: http://psych.fullerton.edu/mbirnbaum/psych466/.

Books by Janetzko (1999) and Janetzko et al. (2002) also include much useful information on how to execute psychology experiments via the WWW. The latter is ala edited book with useful treatments of methodology and statistics, as well as many interesting examples of experiments written by authors of the chapters, ready to run from the accompanying CD. Unfortunately, these works are available only in German at this time.

It is worth mentioning that psychologists in German-speaking nations took to Web research very quickly. The first edition of Internet for Psychologen (Batinic 2000) appeared in 1997, and the German On-line Research Society began its annual meetings that year (Batinic et al. 1999). Within two years, the society invited papers in English as well as German (e.g., Birnbaum 1999c), and recently, much of their work has become available in English (Batinic et al. 2002, Reips 2001b, Reips & Bosnjak 2001).

Client-Side Programming

The "server" is the computer that stores Web pages and delivers ("serves") them to people who send requests to view them. The "browsers," programs that request and display Web pages, or the computers that run programs to contact the server, are known as the clients. Client-side programs run on the participant's computer rather than on the server. They are often implemented by a "plug-in," an extra software component that helps Web browsers run such programs. Modern Web browsers, such as Internet Explorer, Netscape Navigator, and Mozilla (among others), come with JavaScript and Java installed, which are the two most popular languages for client-side programming. These languages are "free," in that neither programmer nor participant needs to buy anything.

Authorware programs, which also run on the client computer, are executed by means of the Authorware Player, a plug-in that can be downloaded free; however, the Authorware program (used to create experiments, demonstrations, or other content) is expensive.

JavaScript is a programming language that has many uses in Web-based research (Birnbaum 2001a). JavaScript programs can be included as source code in a Web page, and the program loads and runs on the client (i.e., the participant's) machine. By running on the client computer, the program does not burden the server of network with computation and communication delays, which might slow down interactions within an experiment. Including source code in the Web page makes it easy for other researchers to study the programs used and to build upon them. Commercial programmers do not want people to be able to copy and reuse their work, but academic scientists thrive when knowledge is available for inspection, criticism, and modification.

A potential problem with running a program on the client's computer is that one relies on the participant to have the proper software installed and running. If a person did not have a compatible browser with the proper plug-ins, then client-side programs would not work for that user. Indeed, many users turned off JavaScript and Java when these programs first became available, fearing security gaps. Today, such fears have mostly been allayed by good experiences with the programs and concerns have been outweighed by the value they provide. Although one now expects to find JavaScript and Java on most …