Probing the boundaries of ignorance in policy analysis.(Initiating Change: Theory and Practice)

Charles E. Lindblom's Wherrett lecture, "Modes of Social Inquiry," along with his book Inquiry and Change (Lindblom, 1990), offer a bold assessment of the unfulfilled theoretical promises and disappointing practical results of the social sciences. The social sciences, argues Lindblom, are deeply troubled; they should and must be transformed, along with the society that spawned them, because they represent little more than a ratification of conformity. The trouble is that social scientists, whether they see themselves as contributors to the development of discipline-based theories, or as policy analysts committed to the alleviation of practical problems, are governed by a wrong-headed but robust notion: The stock of knowledge created by economics, political science, sociology, and other disciplines must be present before we can engage in effective social problem solving. On the contrary, reasons Lindblom: Social problem solving is not a process of applying knowledge so much as a process of probing what to do in the presence of ignorance (see Lindblom, 1990, chap. 4, pp. 29-44).

Ignorance, argues Lindblom, is a product of cognitive impairment, which limits the probes of members of the lay public and their representatives, no less than the probes of social scientists. Cognitive impairment, in turn, is a result of the conformity imposed on members of government bureaus, corporations, families, churches, schools, and social science disciplines. Cognitive impairment blinds us to the need for probing the definition of social problems as well as their possible solutions. "The origin of a social problem lies in the probes that declare it to be a problem" (Lindblom, 1990, p. 36).

This analysis builds on Lindblom's critique by presenting a methodology for probing the structure of complex problems. This probative methodology, which can move policy analysis from a state of cognitive impairment toward one of cognitive empowerment, differs from standard methods of conducting social science research and policy analysis. Standard methods of policy analysis mistakenly assume that the boundaries of complex problems have been adequately defined prior to the analysis of potential solutions for these problems. This mistaken assumption is perhaps best exemplified by the use of cost-benefit analysis. Cost-benefit analysis, although it comes with explicit instructions to be comprehensive, supplies no explicit procedures for generating the knowledge required to succeed in this task. The generation of knowledge required to define the boundaries of a problem is treated as a matter of classification; we are enjoined to enumerate all classes of costs and benefits that should be considered in defining a problem and investigating its solution. In principle, benefits and costs can be efficiently sorted into standard analytic categories: present and future, direct and indirect, internal and external, real and pecuniary. In practice, these boundaries appear indeterminate and unmanageably huge.

Fixed classification schemes are inappropriate and even pernicious when we face real-world problems characterized by ambiguous or conflicting (or simply unknown) goals, a seemingly infinite set of potentially useful policy instruments for achieving those goals, and temporally ordered phases between which feedback and feed-forward loops unexpectedly occur at any time. In this context, it is plainly implausible that any classification of benefits and costs, however comprehensive and rational from an economic standpoint, will supply an adequate definition of the boundaries of a complex problem.(2)

Lindblom's probes represent a form of reasoning best described as erotetic rationality. Erotetic rationality is a more appropriate normative model of reasoning about complex problems than the confirmatory rationality that typifies much of policy-oriented economics, political science, and sociology.(3) Reasoning that is erotetic, or questioning and answering, is plainly different from modes of reasoning that assume that we can offer sound policy advice without first probing the boundaries of a complex problem. Typically, such advice is a product of methodologies designed to confirm some existing knowledge claim rather than generate new knowledge in the face of ignorance.

In arguing the case for erotetic rationality, I examine unbounded ignorance as the central characteristic of ill-structured problems, showing why the process of problem structuring takes priority over the process of problem solving in policy analysis. Then I evaluate alternative methods for structuring problems, focusing on their capacity to provide estimates of the boundaries of our ignorance in defining problems. I conclude by applying a probative methodology that permits us to test for the commission of Type III errors in policy analysis: formulating the wrong policy problem by incorrectly accepting the false hypothesis that there is no difference between the boundaries of the problem, as defined by the analyst, and the actual boundaries of that problem? This probative methodology, based on erotetic rationality, affirms that the discovery of boundaries of ignorance, and not the application of usable knowledge, is a precondition for moving from cognitive impairment to cognitive empowerment in the social sciences and policy analysis?

UNBOUNDED IGNORANCE

Unbounded ignorance in policy analysis is similar to the problem situation facing the homesteaders described by Morris Kline.(6) The homesteaders, while clearing their land, are aware that enemies lurk in the wilderness that lies beyond the clearing. To increase their security, the homesteaders clear a larger and larger area, but they never feel completely safe. They frequently must decide whether to clear more land or attend to the crops and domesticated animals within the perimeter. The homesteaders do their best to push back the wilderness, anxiously aware that the wilderness is always there and that one day enemies may surprise and destroy them. The homesteaders also are aware that there are enemies within, enemies who may undermine and distort their judgments. They hope that they will not choose to tend the crops and livestock when they should have chosen to clear more land.

Planners and policy analysts, like allegorical homesteaders, require methods that enable them to know when they have cleared enough land, thereby approximating the proper boundaries of a problem. Most available policy-analytic methods, however, assume that the boundaries of complex problems have already been defined. The prevailing view, accepted almost by default, is that problems come in relatively well-bounded packages that include an explicit statement of some decision maker's preferences, an identification of the alternatives available for satisfying these preferences, and a model that relates the alternatives and the preferences in a manner permitting an efficient choice among the alternatives.(7)

This view of policy problems assumes that the boundaries circumscribing preferences, alternatives, and their relationships have been satisfactorily defined. Here, the analyst is faced with what Mitroff characterizes as a structured decision problem, where the relationships between decision makers (Di), preferences or utilities (Uij), alternatives (Ai), outcomes (Oj), and states of nature (Sj) are certain, probabilistic, or uncertain. A structured decision problem is one about which enough is known that it can be formulated in ways that permit the use of methods that yield precise estimates (see Mitroff, 1974, pp. 223-224). By contrast, an ill-structured problem is one that is squishy, messy, or wicked.(8) Formally, an ill-structured decision problem is one for which decision makers (Di), preferences or utilities (Uij), alternatives (Ai), outcomes (Oj), or states of nature (Sj) are unknown or profoundly equivocal.

Ill-structured decision problems assume a single or unitary actor exercising reasoned individual choices. Processes of policy formation, however, involve collective choices shaped by multiple actors. In this context, an ill-structured policy problem is one with at least four distinct characteristics (cf. Harmon & King, 1985, p. 28):

* Policy goals. The goals of a policy are ambiguous or unknown, so that determining what goals to achieve is part of the problem.

* Policy phases. The phases through which policy goals are pursued are indeterminate.

* Policy instruments. The policy instruments required to achieve goals are ambiguous or unknown.

* Policy problem domain. The domain of potentially relevant goals, phases, and instruments is unbounded.

Ill-structured problems originate in competing mental representations of problems that are created, maintained, and changed by individuals and groups with a stake in policy making. These mental representations are not confined, temporally or spatially, to phases of policy making conventionally labeled problem formulation or agenda setting. Instead, competing problem representations arise in every phase and are distributed throughout the entire process. Stakeholders situated at various points and locations in this process actively construct, on the basis of their own interests and experience, different representations of problems. Differences among problem representations are based on contrasting assumptions about those external conditions, which John Dewey called a "problem situation." Contrasts among organized sets of …