A test of the cognitive load hypothesis: investigating the impact of number of nonverbal cues coded and length of coding session on observer accuracy.(Report)

In their meta-analysis investigating the relationship between extraversion and nonverbal behavior La France, Heisel, and Beatty (2004) found a substantial negative correlation between effect size and sample size, which they explained using the cognitive load hypothesis. The cognitive load hypothesis predicts that increases in coding scheme complexity result in greater opportunities for observer error. To test this hypothesis, the impact of coding scheme complexity on observer error was assessed via varying the number of nonverbal cues coded and the length of observational coding session. The decision to increase the number of nonverbal cues observers coded created 26% more errors, and over time observers made 10% more errors.

Keywords: Coding Scheme; Cognitive Load; Meta-analysis; Nonverbal Communication

**********

In their meta-analysis investigating the relationship between self reported extraversion and observer reports of nonverbal codes associated with extraversion (e.g., eye contact, proximity, gestures, smiling, etc.), La France, Heisel, and Beatty (2004) found a substantial negative correlation between effect size and sample size (r = -.40). They argued that this relationship may be an indicator of an underlying methodological artifact, which they termed cognitive load. Their cognitive load hypothesis posited that as coding schemes become increasingly complex observer error increases as well. Increasing sample size, they argued, contributes to the complexity of coding communicative behavior, which leads to more error and attenuated effect sizes. They also provided evidence revealing negative correlations between sample size and effect size in other meta-analyses (average weighted r = -.35). (1)

Quantitative content analysis is an important method with increasing popularity among social scientists including communication scientists (Riffe, Lacy, & Fico, 1998). Perhaps not surprisingly, over time classification systems have become more complex. For example, earlier studies investigating the relationship between nonverbal communicative behavior and personality variables had observers code relatively few (i.e., five or fewer) nonverbal behaviors (Kendon & Cook, 1969; Mallory & Miller, 1958; Mobbs, 1968; Pedersen, 1973; Ramsay, 1966; Steer, 1974). More recent personality and nonverbal communication research, however, has increased the number of nonverbal cues observers are counting. For example, Lippa (1998) had observers rate over 30 nonverbal behaviors. Riggio and Friedman (1983) had raters code 29 nonverbal cues and in a later study had coders record incidences of 27 nonverbal behaviors (Riggio & Friedman, 1986). Berry and Hansen (2000) had coders rate 17 nonverbal behaviors. Although the decision to have observers code many rather than few behaviors may be consistent with hypothesis testing, it comes at a methodological--and theoretical--price. Indeed, Riffe et al. note that as coding schemes increase in their complexity coder errors are likely to increase (p. 107). The current study was designed to test this assertion.

Types of Content

Burgoon and Baesler (1991) define microscopic nonverbal behavior measurement as the observation of single concrete behaviors that are event-based or that occur during relatively short time intervals. Alternatively, macroscopic measurement typically involves a compilation of nonverbal behaviors that are more abstract and occur over an extended time period or events. They state that there are conceptual and methodological benefits and consequences to each type of measurement. Riffe et al. (1998) distinguish between manifest content and latent content in content analysis. Manifest content refers to easily recognized phenomena--phenomena that can be counted easily. For example, noticing whether a speaker uses a vocal segregate (e.g., um) would be easily recognized by an observer. By contrast, latent content exists when "the meanings embedded in the content [must be] interpreted by some observer" (p. 107). Asking observers to determine the degree to which a speaker is extraverted, psychotic, or neurotic for example requires that coders recognize and interpret communicative behavior. The lens through which such interpretation occurs includes a variety of assumptions that are specific to the individual. Riffe et al. argue that analyzing latent content is more complex and often times leads to more interrater disagreements.