AccessMyLibrary provides FREE access to millions of articles from top publications available through your library.
* Abstract Advantages and disadvantages of Web and lab research are reviewed. Via the World Wide Web, one can efficiently recruit large, heterogeneous samples quickly, recruit specialized samples (people with rare characteristics), and standardize procedures, making studies easy to replicate. Alternative programming techniques (procedures for data collection) are compared, including client-side as opposed to server-side programming. Web studies have methodological problems; for example, higher rates of drop out and of repeated participation. Web studies must be thoroughly analyzed and tested before launching on-line. Many studies compared data obtained in Web versus lab. These two methods usually reach the same conclusions; however, there are significant differences between college students tested in the lab and people recruited and tested via the Internet. Reasons that Web researchers are enthusiastic about the potential of the new methods are discussed.
CONTENTS INTRODUCTION Computer Developments that Set the Stage TECHNIQUES Getting Started: HTML Forms Client-Side Programming Server-Side Programming Run Your Own Server Stimulus Delivery METHODOLOGY Multiple Submissions Dropouts in Between-Subjects Research Recruiting Participants Via the Web Sampling Bias and Stratified Analysis Response Bias Experiment or Bias PILOT TESTING IN THE LAB WEB VERSUS LAB SUMMARY AND CONCLUSIONS
In the last decade, a new protocol for sending information on the World Wide Web (WWW), hypertext transfer protocol (HTTP), created a new way to conduct psychological research. This new technique allows researchers to collect data from participants all over the world 24 hours a day and seven days per week. Surveys and experiments can be delivered quickly to anyone connected to the Web and data can be saved automatically in electronic form, reducing costs in lab space, dedicated equipment, paper, mailing costs, and labor. Once an experiment or survey is properly programmed, data can be stored in a form ready for analysis, saving costs of data coding and entry that used to be an expensive and time-consuming part of the research process.
Computer Developments that Set the Stage
By early 1995, a number of changes were under way that created the conditions for a new approach to psychological research:
* Computer hardware and software improved; it became less expensive to own computers and easier to operate them.
* The number of individuals who owned or had access to personal computers, email, and the WWW was increasing exponentially.
* More and more valuable content was being added to the WWW.
* Hypertext markup language (HTML) 2 supported the technique of forms.
The new standard of HTML 2, introduced in late 1994, supported forms, which allowed a person viewing a Web page to easily send data back to a designated server, which could process, code, filter, and save data electronically. This technique made it possible for a person, even without an email account, to be able to participate in a survey from an Internet-connected computer (e.g., in a public library or campus computer facility) even if that computer was not configured to send email.
Within a few months, psychologists began using this technique to collect data in surveys and experiments. A number of these "pioneers" contributed chapters to Birnbaum's (2000b) edited book, which included a chapter by Musch & Reips (2000) that summarized the (short) history of psychological experiments (not surveys) on the WWW [see also Musch (2000)].
Musch & Reips (2000) noted that the first psychological experiments (with manipulated variables) conducted via the Web were those of Welch & Krantz (1996) and Krantz et al. (1997). Krantz (1998) maintains a Website, "Psychological Research on the Net," (http://psych.hanover.edu/research/exponnet.html) that lists experiments currently running on the Internet. The site had 35 links to on-line studies on June 17, 1998. By May 11, 1999, there were 65 links, and on May 10, 2003, there were 150 links, including 45 to projects in social psychology and 30 to studies of cognitive psychology.
Several of these links lead to multiple experiments. For example, Jonathan Baron (http://www.psych.upenn.edu/~baron/qs.html) has been running about 50 studies per year from his site. The PsychExps site at Ole Miss (http://psychexps. olemiss.edu/) listed 33 lab and research studies on May 10, 2003, most of which were in cognitive psychology. Similarly, the student research project site at my Decision Research Center of Fullerton (http://psych.fullerton.edu/mbirnbaum/ decisions/thanks.htm) listed 21 studies on that same date. Ulf-Dietrich Reips's Web Experimental Psychology Lab (http://www.psychologie.unizh.ch/genpsy/Ulf/Lab/ WebExpPsyLab.html), which began operations in 1995, had links to 16 active studies in English and 12 in German plus about 80 in its archive on this same date in 2003. The Web Experiment List (http://genpsylab-wexlist.unizh.ch/) listed 166 studies in 2003. Although not all studies are listed in these sites (and some items are duplicates), the numbers give an indication of the rapid expansion of this method of doing research. A few hours visiting the links would convey to the reader a general impression of the kinds of studies being done this way.
Although any given work may have more than one focus, I divide my review in three distinct topic areas, according to my judgment of a work's main thrust. These three areas are (a) Techniques that describe of analyze "how-to" issues. For example, how can we randomly assign participants to between-subjects conditions? (b) Web methodology, which deals with internal and external validity of Internet research. For example, what are the threats to internal and external validity if people drop out of Web-based experiments, and what are the considerations of using particular techniques to reduce such dropouts? (c) Web versus lab comparisons, which includes findings specific to the Internet and comparisons of Web and lab studies of the same phenomena. For example, do experiments in decision making that are conducted via the Web yield the same conclusions as those done in the lab? In what ways do results obtained in Web and lab differ?
A fourth major area of research involves social psychology of the Internet, which deals with the Internet as a new social situation or communication medium. For example, do people exhibit different social behaviors on-line and in person? This topic is the focus of a separate review (Bargh & McKenna 2004) and will not be covered here except in relation to methodological issues in Web research.
Getting Started: HTML Forms
Probably the easiest way to get started with the techniques of Web-based research is to make a survey or experiment using one of the free programs (e.g., Birnbaum 1998) to create the Web page for a simple survey of factorial experiment. These programs are available from the following URL: http://psych.fullerton.edu/ mbirnbaum/programs/.
SurveyWiz and FactorWiz are Web pages that make Web pages for collecting data via the WWW. Within each Web page is a list of instructions for use (see also Birnbaum 2000c, 2001a). Creating a Web form with SurveyWiz is as simple as typing the questions and pushing buttons for the type of input device desired. The program supports text boxes, which are boxes into which a participant can respond by typing a number or short answer. SurveyWiz and FactorWizRB also support radio buttons, which allow the user to click along a rating scale.
The most fundamental technique of Web-based research is HTML, the formatting language used to compose Web pages. A Web page (a document) can contain formatted text, links to other information on the Web, pictures, graphics, animations, and other media such as sounds or video. Many software products are available to create Web pages, including free ones such as Birnbaum's (2000c) FactorWiz, WEXTOR by Reips & Neuhaus (2002), Schmidt's (1997b) WWW Survey Assistant, and White & Hammer's (2000) Quiz-o-matic (which makes self-scoring quizzes). Despite the availability of free and commercial software to make Web pages, a Web researcher needs to have a basic understanding of the "tags" (commands) of HTML. There are many free tutorials available on the WWW for learning HTML, as well as many commercial books on the subject.
Books by Janetzko (1999) and Janetzko et al. (2002) also include much useful information on how to execute psychology experiments via the WWW. The latter is ala edited book with useful treatments of methodology and statistics, as well as many interesting examples of experiments written by authors of the chapters, ready to run from the accompanying CD. Unfortunately, these works are available only in German at this time.
It is worth mentioning that psychologists in German-speaking nations took to Web research very quickly. The first edition of Internet for Psychologen (Batinic 2000) appeared in 1997, and the German On-line Research Society began its annual meetings that year (Batinic et al. 1999). Within two years, the society invited papers in English as well as German (e.g., Birnbaum 1999c), and recently, much of their work has become available in English (Batinic et al. 2002, Reips 2001b, Reips & Bosnjak 2001).
Authorware programs, which also run on the client computer, are executed by means of the Authorware Player, a plug-in that can be downloaded free; however, the Authorware program (used to create experiments, demonstrations, or other content) is expensive.