When two factors don't reflect two constructs: how item characteristics can produce artifactual factors.

Many scales that assess variables of interest to organizational researchers (e.g., employee personality) contain items written to reflect both ends of the continuum of the construct of interest. Factor analyses of such scales sometimes suggest the existence of two factors, each containing items written in one direction. When assessing a scale that contains oppositely worded items that form separate factors, it is important to ascertain whether the scale measures one or two constructs. The seeming independence of oppositely worded items can be caused, not by underlying constructs, but by the way in which people respond to items.

In this paper, we first review literature showing that item direction factors within scales is widespread. Next, we discuss how people's patterns of responses to oppositely worded items can produce two-factor structures. The existence of these patterns is demonstrated in Study 1 with job satisfaction data from a sample of employed subjects. The production of factors based on response patterns to different types of items is shown with simulated data in Study 2. Finally, we present guidelines for determining when two-factor structures are likely artifactual.

Item Type Factors In Existing Scales

Factors produced by items phrased in opposite directions have been noted for several scales in the organizational domain. Such factors have been found in Hackman and Oldham's (1975) Job Diagnostic Survey (Idaszak & Drasgow, 1987), Meyer and Allen's (1984) Affective and Continuance Commitment Scales (Magazine, Williams & Williams, in press), Rizzo, House and Lirtzman's (1970) role ambiguity and conflict scales (McGee, Ferguson & Seers, 1989), and Spector's (1988) Work Locus of Control Scale (Spector, 1992). In all cases these factors were interpreted to be artifacts of wording direction. However, there has been limited work done on the underlying mechanisms that produce artifactual factors.

Several possible mechanisms have been suggested for artifactual factors, including lack of ability to understand negatively worded items (Cordery & Sevastos, 1993) and carelessness in reading items (Schmitt & Stults, 1985). In addition Schreisheim and his colleagues noted that negatively worded items are often less reliable and valid than positively worded items (Schriesheim, Eisenbach & Hill, 1991; Schriesheim & Hill, 1981), possibly due to the greater difficulty for people to interpret negatively worded items. All of these researchers have recommended that care be taken in mixing negatively and positively worded items in the same questionnaire.

We discuss in this paper how the distribution of item responses can produce direction factors in some scales. This mechanism is not necessarily responsible for item direction factors in all of the scales noted. For example, it seems quite likely that the Schmitt and Stults (1985) carelessness response explanation is correct for the JDS. However, not all scales produce negatively worded items by negating positively worded items as does the JDS. Our mechanism does not rely on item negation (adding "not" to a positively worded item, e.g., "I do NOT like my job"). It is most likely to occur with scales that assess affect or personality because of the tendency to use oppositely worded items that in some cases are rarely endorsed by subjects.

In the affect domain, item direction factors have often been interpreted as reflecting multiple constructs. For example, items that reflect negative versus positive emotions have been interpreted as assessing the independent constructs of negative and positive affectivity (Brief & Roberson, 1989; Burke, Brief & George, 1993; George, 1991; Watson, Clark & Carey, 1988; Watson & Tellegen, 1988). However, Gotlib and Meyer (1986) argued that the two-factor structure of the Multiple Affect Adjective Checklist was likely caused by response sets. Scheier and Carver's (1985) dispositional optimism scale (the LOT) has been interpreted as containing two independent dimensions - optimism and pessimism (Marshall, Wortman, Kusulas, Hervig & Vickers, 1992). Pilotte and Gable (1990) suggested that the two factors they found with their scale of computer anxiety might reflect two constructs.

Item direction factors have been found in several scales, but there are many scales that do not produce this effect. For example, the phenomenon has not been noted for job satisfaction scales. The Job In General Scale has been found to be clearly unidimensional (Ironson, Smith, Brannick, Gibson & Paul, 1989). The nine facet Job Satisfaction Survey did not form item direction factors even though approximately half the items were phrased in each direction (Spector, 1988). With scales that produce the item direction factors, there are several possible mechanisms that might cause them.

Although the item direction factors in scales could be produced by independent constructs, it seems more likely that they are artifacts. An examination of the literature on the psychometric properties of item responses suggests that the way in which people respond to items that vary in direction can produce item direction factors even when the underlying construct is unidimensional. Nunnally and Bernstein (1994) noted that pseudofactors can emerge when items with different directions or distributional properties are factor analyzed. Van Schuur and Kiers (1994) argued that factor analysis is inappropriate for many scales because of how people respond to them. We next discuss how such responses can produce factors based on direction of item wording.

How Responses To Items With Different Characteristics Can Produce Artifactual Factors

In computing total scores with scales that contain oppositely worded items, negative items are reverse scored so that strong disagreement is given the same number as strong agreement with positive items. Therefore, the scoring of disagreement with the first item type will be equivalent to agreement with the second. The assumption underlying such a scoring method is that individuals who disagree strongly with an item on one side of the continuum will usually agree with items on the other side. This should result in a similar magnitude of correlations between oppositely and similarly worded items. If similarly worded items intercorrelate around .40, then oppositely worded items should intercorrelate around -.40. The uniformity in correlation magnitude across a set of items that assess the same construct should produce a single factor. If a subset of items based on wording direction intercorrelate more highly with one another than with oppositely worded items, a two-factor structure will emerge even in a set of items that assesses a single construct.

Although we assume that reverse scoring oppositely worded items makes the item scores equivalent, this is not necessarily the case. As pointed out by Thurstone (1928) in his classic paper on attitude scale construction, a person will tend to agree with items that are close to his or her attitude and disagree with items that are far from it in either direction. This idea that people agree with items that are close to their level on the trait and disagree with all others is termed the ideal point (Cliff, Collins, Zatkin, Gallipeau & McCormick, 1988) or the unfolding principle (Andrich, 1988). This implies that there is a curvilinear, rather than linear, relation between the extent of agreement with an item and the person's standing on the construct of interest. Given a particular item, people whose standing is either higher or lower than the item will be likely to disagree with …