Deconstructing research on systems of care for youth with EBD: frameworks for policy research.(emotional and behavior disorders)

Integrated service systems of care for children with severe emotional disturbance based on the systems-of-care framework have historically been viewed and studied as programmatic and clinical interventions. Consequently, researchers have most commonly applied program evaluation methods to study their effectiveness. These methods rely largely on quasi-experimental designs with repeated outcome measures. Systems of care can also be understood, however, as a set of systems-level policies and principles designed to alter the delivery of services provided to youth and their families. From this perspective, researchers can apply analytical and methodological tools traditionally used in health policy analysis to study the multiple levels of systems-of-care impacts. This article examines how these and other methods for policy analysis can be utilized to measure the impact of the systems-of-care approach. Where possible, results from existing systems-of-care evaluation studies illustrate how these analyses and methods can be practically employed.

**********

THE REFORM OF HUMAN SERVICE DELIVERY systems is a complex endeavor with a not entirely encouraging history, as past evaluation efforts have demonstrated their marginal impact on improving outcomes for children and families (Kagan & Neville, 1993). The challenges are magnified when the reform process is wide ranging and involves a myriad of goals, such as the movement to create systems of care for youth with severe emotional disturbances (SED; Stroul & Friedman, 1986). Spearheaded by the National Institutes of Mental Health (NIMH) and the Center for Mental Health Services (CMHS), the principles and objectives of systems of care require sweeping changes in service systems. These changes range from the alteration of service delivery approaches to fundamental shifts in the role of care providers. Research on such comprehensive reform efforts has produced contradictory and at times controversial results (Rosenblatt, 1998). In this article we provide a health services research and policy analysis perspective on understanding the aspects and impacts of the systems-of-care policy reform movement.

Although a range of health services research techniques have been used to study systems of care, the primary focus in the scientific debate to date has been whether these systems have an impact on the health status of the youth and families being served. Typically, researchers have utilized program evaluation and clinical research designs to compare systems of care to traditional service delivery systems. These studies provide rich and varied data sets that have been explored through a variety of analytical techniques (e.g., Bickman, 1996a, 1996b; Rosenblatt, 1998; Stroul & Friedman, 1996), consistently demonstrating that the systems-of-care reform process yields systemic changes in service delivery. However, the children's mental health research field currently finds itself in a quandary, described by Farmer (2000), in which convincing evidence regarding the effectiveness of these reforms at the individual child and family levels is lacking.

Conducting research on effectiveness of systems of care is fundamentally complex (Farmer, 2000; Rosenblatt, 1998). Complexity is everywhere: in the nature of the service system reform, the children and families served, and the research questions posed. Such complexity, however, is not unique to the systems-of-care movement. Historically, health services researchers have tackled similar problems, ranging from the study of the effectiveness of managed care to specific medical procedures. In addition, with few exceptions, the full range of mental health treatments for youth lack a preponderance of effectiveness evidence in real-world settings. Consequently, recent initiatives have emphasized the need to distinguish between efficacy research of treatments implemented under controlled settings and effectiveness research of treatments implemented in real-world clinical environments (National Advisory Mental Health Council, 1999). In this article we address the fundamental problem of the complexity of the role of research in the policy process by applying three key concepts from health services research and policy analysis to a current three-part framework of service delivery in systems of care. Examples of different approaches to conducting research within this model are provided, and the implications of the model for the development of systems of care and children's mental health policy are discussed.

THREE LEVELS OF ANALYSIS FOR SYSTEMS OF CARE

At its core, the systems-of-care approach is a systemic, policy-oriented change in the structure and delivery of services. Although little convincing empirical data exists concerning the effectiveness of these systemic reforms, compared to traditional practices, on the functional status of youth being served, the systems-of-care approach is an extraordinarily popular form of service delivery in the United States. CMHS has provided grants to 67 states, communities, and tribal nations for the implementation of systems of care (Holden & Brannan, 2002), and states from California to Vermont have implemented the reforms statewide in their public mental health systems. There is a rich and growing descriptive research base on systems of care, providing important evidence in shaping policy decisions and understanding service delivery (e.g., Brannnan, Baughman, Reed, & Katz-Leavy, 2002; Holden, De Carolis, & Huff, 2002; Manteuffel, Stephens, & Santiago, 2002; Rosenblatt, 1998; Stroul, Pires, Armstrong, & Zaro, 2002).

An understanding of the multiple levels of a system of care is essential to developing a framework for policy-relevant research. By any definition, a system of care is a complex strategy for system reform (see Stroul & Friedman, 1986, 1996). Although human service systems can be analyzed from a wide range of perspectives, current research on systems of care tends to focus on three levels of analysis: (a) the systems level, (b) the programmatic level, and (c) the practice level (Rosenblatt, 1998).

The systems level refers to the structure, organization, and financing of services. There are many examples of systemic reform in health services. For example, managed care is predominantly a reform that occurs at the systems level, where fiscal changes (such as capitation) and structural changes (such as utilization review) are put in place to provide more efficient service delivery. Over the past decade, systemic reform in the delivery of children's mental health services has been promulgated largely through the systems-of-care approach (Stroul & Friedman, 1996). The systems-of-care model emphasizes many systems-level alterations, including developing linkages among child-serving agencies (e.g., mental health, juvenile justice, child welfare, and education), using community-based care in lieu of restrictive placements, developing a continuum of services, restructuring service financing (i.e., blended funding pools), and creating interagency policy and treatment teams for coordinated care.

Systems of care are also composed of program-level interventions that can include traditional clinical services (such as outpatient and inpatient care) or more innovative, integrated services (such as therapeutic foster care, case management, and special day-school programs). Regardless of the level of innovation at the program or system level, the ultimate success of any care is at least in part dependent on what occurs at the practice level, which is the ways in which care providers interact directly with children, their families, and their support systems. Practice-level interventions in a system of care include a range of traditional, office-based psychotherapeutic approaches, such as cognitive--behavioral therapy, family therapy, and play therapy, as well as less traditional models such as multisystemic therapy (MST; Henggeler & Borduin, 1990; Henggeler, Melton, & Smith, 1992).

Paradoxes occur, however, in conducting research across the three levels of the systems-of-care approach. The practice-and program-level interventions--for which there are significant efficacy and even some effectiveness data--are not being embraced and adopted by communities in the same manner as the systems-of-care approach, for which there is relatively less effectiveness data (National Advisory Mental Health Council, 1999; U.S. Department of Health and Human Services, 1999). The reasons behind this paradox are many, and they include the perceived and actual relevance of the research conducted on practice- and program-level interventions to the real world. Understanding this paradox is essential, however, for furthering the field of children's mental health services and policy research.

THREE KEY COMPONENTS OF HEALTH SERVICES RESEARCH

The knowledge typically generated by health services research regarding the impacts of systemic, programmatic, or clinical …