The illusion of technique: how method-driven research leads welfare scholarship astray *.

When federal lawmakers ended "welfare as we knew it" in 1996, state officials had to generate a host of new rules and regulations to govern the Temporary Assistance for Needy Families (TANF) program. Their efforts produced a collection of state TANF programs that differ today in a remarkable variety of ways. The new article by Gordon De Jong and colleagues aims to bring order to this complex policy landscape and to shed light on the decision-making dynamics that have created it. We second the authors' call for a systematic analysis of how TANF rules have varied across states and over time. We also share their desire to learn which TANF rules "hang together" in meaningful ways and whether these clusters suggest specific geographic, programmatic, or temporal patterns of development. Nevertheless, we disagree with how the authors pursue these goals and see good reasons to be skeptical of their conclusions.

Our article focuses on De Jong et al.'s factor analysis of TANF rules. The policy dimensions generated by this analysis are the cornerstone of all that follows. They underlie the authors' findings on policy diffusion as well as welfare migration (De Jong, Graefe, and St. Pierre, 2005), and they are the products from this study that are most likely to find direct use in subsequent scholarship.

Like any piece of empirical research, the article by De Jong et al. reflects a particular way of combining method, theory, and substantive knowledge. We believe the authors have negotiated this triad in an unbalanced way that privileges statistical priorities over other relevant concerns.

Our intention is not to suggest that statistical considerations are insignificant, but rather to underscore that researchers confront tradeoffs in their efforts to balance theory, substance, and method. Rather than face up to these tradeoffs, De Jong et al. bypass them. The authors adopt a method-driven approach (Shapiro, 2005), handling key analytic choices as if they were merely technical matters, neutral with regard to theory and substance. This "illusion of technique" is a wishing away of tough choices. Steadfast reliance on technical decision rules has not put the authors' empirical conclusions on more neutral ground; it has put them into question.

A Method-Driven Analysis

To make their dimensions useful for the study of welfare migration (De Jong, Graefe, and St. Pierre, 2005), the authors limit their analysis to rules that define the stringency of TANF provision "from the clients' perspective" (2006: 760). Employing the Urban Institute's Welfare Rules Database (WRD) as their sampling frame, they then reduce their policy set in two stages. First, they drop all rules that do not exhibit at least 10 percent variation across states. Second, they drop 35 of the remaining 78 rules because these items correlate weakly with other measures in the analysis.

At each decision point, the authors bypass theory and substance, opting instead for the most direct route to a tractable and statistically advantageous set of welfare rules. In the process, they include some TANF rules that are incidental to their research questions and exclude others that should be central to their analysis. In the three sections below, we follow the points in our triad of research priorities, describing how the authors' technical decision rules create problems for their substantive portrayal of welfare reform, for their testing of the race to the bottom thesis, and, ultimately, for their use of exploratory factor analysis.

Substance: Creating a Portrait of Welfare in the States

In studies of welfare migration (De Jong, Graefe, and St. Pierre, 2005), there is a statistical logic to ignoring TANF rules with less than 10 percent variation across states. If one is assessing policy change, however, this decision rule is more questionable. Widely employed policies are often among the most important. For example, when federal officials created TANF they told states to choose whether noncitizens who arrived before 1996 would be eligible for its benefits. When every state except Alabama opted to make this group eligible, their decisions had profound consequences for poor immigrants (Fix and Zimmerman, 2002). Yet because …