How will we judge whether welfare reform has been a success? When taxpayers, the federal government and state policy-makers say they want to see "results" from the new welfare programs, what do they mean? The answers to these questions are critical because definitions of success can determine current and future policy. If the measures chosen are too narrow (or too broad), public policy could be skewed and new programs could hurt the people they are intended to help.
In this report, a group of senior state welfare officials from throughout the Midwest look beyond reductions in the welfare rolls and identify ten criteria that can be used to understand the impact of new policies on children and families.
What is WELPAN?
The Midwest Welfare Peer Assistance Network (WELPAN) is a group of senior welfare officials who began meeting in October 1996 to discuss the complex policy and administrative issues associated with welfare reform. WELPAN members come from seven states--Illinois, Indiana, Iowa, Michigan, Minnesota, Ohio and Wisconsin--whose governments represent the broad political spectrum.
In the past, decisions on welfare policy generally flowed from the federal government to the states. Now states are making more decisions and the primary innovations in welfare policy are occurring outside of Washington, D.C. WELPAN brings senior state administrators together to encourage cross-fertilization of ideas and strengthen reform efforts.
WELPAN is coordinated by the Family Impact Seminar (FIS), in Washington, D.C., and the Institute for Research on Poverty (IRP), at the University of Wisconsin-Madison. FIS is an information and resource center on children and family policy. IRP is a university-based research organization that analyzes the causes of and solutions to poverty. The network is funded by The Joyce Foundation, which is based in Chicago.
This report is the first of a series. Although it grows out of discussions among WELPAN participants, it should not be construed as representing the views of any particular members of the network, nor of the states they represent.
Charlene Burkett-Sims, Indiana Family and Social Services Administration
|For more information:
FAMILY IMPACT SEMINAR
1730 Rhode Island Avenue NW
Washington, D.C. 20036
(202) 496-1964 ext 12
Report can be downloaded from these websites:
Funded by the Joyce Foundation
WELPAN Staff Coordinators
Elisabeth Boehnen, Institute for Research on Poverty, University of
*Asterisk denotes those individuals who participated in the early meeting discussions about measures of success but who have not reviewed this report.
This is a time of unprecedented policy change. The replacement of the federal Aid to Families with Dependent Children (AFDC) entitlement program with the Temporary Assistance for Needy Families (TANF) block grants to states has radically changed the purpose of welfare from providing cash support to promoting work and other desired activities. In addition, providing states with more flexibility is stimulating a lot of innovation. These shifts raise significant questions for state officials, who are increasingly responsible for shaping welfare policy themselves. How will officials know if the newly-designed welfare policies are working? Will the lives of parents, children and society as a whole be better as a result of these policy changes? What should policymakers do next to build on what's working and fix what's not?
The recent public debate on welfare reform tends to focus on two specific outcomes: reducing the number of people who are on welfare rolls and getting welfare clients into jobs. In this report a group of senior welfare administrators from seven Midwestern states conclude that these two criteria alone are not sufficient to measure the success of welfare reform.
The administrators found that under federal and state legislation, welfare agencies will be held accountable for success in four initial areas: welfare dependency, labor market attachment, economic well-being, and parental financial responsibility. After thinking through the broader social and economic contexts in which welfare reform occurs, the administrators also identified six additional areas in which welfare agencies will ultimately be held accountable: child, parent and family well-being, family formation, community involvement and economic development.
The administrators identify some ways to measure progress in these areas, outline a methodology for choosing measures of success, and define a common language for discussing progress under welfare reform.
This report is not a blueprint for evaluating welfare reform. It is intended to stimulate thinking and guide discussion on this important topic. It is the first in a series of reports from the Midwest Welfare Peer Assistance Network, a group of senior welfare administrators who are on the front lines of welfare reform in Illinois, Indiana, Iowa, Michigan, Minnesota, Ohio and Wisconsin.
The goal of the AFDC program was to provide income support to needy, single parents in order to care for their children. By contrast the goal of the TANF program is to change the behavior of adults and end the entitlement to support. The 1996 Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA), which establishes the TANF block grants, stated that the purpose of the program is to reduce dependence on public assistance by promoting job preparation, work and marriage; reduce out-of-wedlock births; and help poor children remain in their homes.
The recent public debate on welfare reform has primarily focused on measuring two outcomes: reducing the number of people who are on the welfare rolls and getting welfare clients into jobs. Early on in their meetings however, WELPAN members realized that these outcomes are not the only goals for which welfare agencies will be held accountable, by federal law or by others.
A wide range of stakeholders are interested in the results of welfare reform, including government officials, the business community, the general public and advocacy, non-profit and community groups. Some are interested in finding out whether welfare reform will increase or decrease taxes, others in how it will affect teen pregnancy and marriage rates, and others are primarily interested in effects on child well-being. Some are hopeful that as more people leave welfare and enter the work force, substance abuse and violence will decrease and communities will become safe. Still others are most concerned about whether the reforms will result in an increase (or decrease) in homeless families or children being placed in foster care.
The discussion about goals and measures now assumes a greater importance for many state welfare officials. In the past the federal government was primarily responsible for designing welfare programs and defining how they would be assessed. State welfare agencies were held primarily accountable for outcomes that measured program operations. Measures of success, for example, emphasized process goals, such as whether applications for assistance were acted upon in a timely manner and whether cash benefits were calculated accurately.
With welfare reform, however, program goals are coming from multiple directions and focus on behavioral changes in welfare clients, which are much harder to define and measure. In the TANF block grant, the federal government identifies a set of results that states must produce in order to get funding. In addition, each state welfare plan identifies a set of goals, some of which go beyond the federal requirements. Further, advocates for the poor, taxpayers and many other stake-holders to whom agencies are ultimately accountable, have definite opinions about what should be the outcomes of welfare reform. All of these definitions of success converge on the state welfare agencies. WELPAN members found that stepping back to look at all of these directives and thinking comprehensively about the goals of their programs helped them to better understand their own reform efforts. They realized that to design and implement public assistance programs without this step would be like planning a travel itinerary before choosing a destination--the travelers wouldn't recognize the town when they had reached it.
Defining success and a method for measuring progress are also critical because these definitions drive programming. If the measures chosen are too narrow (or too broad), public policy could be skewed and new programs could hurt the people they are intended to help. For example, one goal for welfare reform is to reduce the caseloads of people on welfare rolls. There are many paths to reach this goal. One strategy would be to simply not offer assistance to any more people. But if caseload reduction is considered in the context of other goals, such as increased employment or child well-being, other strategies would seem much more appropriate.
With this report, WELPAN hopes to encourage more discussion about how to define and evaluate the success of welfare reform. The report is organized in three parts:
Part I: Areas of Concern--A Starting Point for Discussion
Part II: A Method for Choosing Measures and Defining Success
Part III: What Do We Mean by Success? Concepts, Terms and Issues
This report is not a blueprint. It is basically a summary and synthesis of the insights from discussions among WELPAN members, who read background papers, heard from expert researchers, shared lessons from demonstrations in their respective states, and engaged in multiple discussions over the course of a year.
The report is designed to stimulate thinking and serve as a starting point for further discussion. WELPAN hopes this guide will be useful for state and local officials, administrators, and front-line staff in welfare agencies in other states as well as those represented in the network. WELPAN also thought it would be useful for the general policy community and those to whom welfare agencies are immediately accountable--legislators and ultimately the general public.
WELPAN members identified four primary areas of concern to summarize how welfare reform will be assessed under the federal 1996 Personal Responsibility and Work Opportunity Reconciliation Act and the welfare plans in members' home states. These are the areas in which the welfare administrators believe they will be held most directly accountable for changing behavior. Worksheet I on page six outlines these areas and breaks them down into more specific measures that can be used to evaluate progress in each area. The measures assess different aspects of the broad concept.
WELPAN members also identified six additional areas of concern, outlined in Worksheet II on pages seven and eight, that they believe many states are using to measure progress under welfare reform. These areas reflect the broader context in which welfare reform occurs and summarize the concerns of many different stakeholders. Some of these outcomes may be regarded as the ultimate and most important measures of welfare reform's success because they reflect the underlying rationale for public assistance programs.
Here is an example. One of the main reasons for requiring parents to work is the belief that, in the long run, their children will benefit economically, socially and psychologically. Even though most Midwestern states have not explicitly established child-related goals, administrators are aware that they will be held accountable if children are found, on average, to be significantly worse off as a result of welfare reform. As a result, many states assume the responsibility to help parents ensure that their children are adequately cared for while they are working. States are putting substantially increased monies into child care subsidies and to increase the supply and quality of day care. As further evidence of the states' concern about the impact of welfare reform on children, six of the seven WELPAN states participated in the early stages of a federally funded project to rigorously evaluate child outcomes in their welfare reform evaluation plans.
Worksheets I and II provide a reasonably comprehensive, though not exhaustive, list of criteria that states might examine to understand and assess the effects of new welfare policies. WELPAN recommends that these lists be used as a starting point for further discussion on this topic. WELPAN members agreed on the importance of each of these areas. Since each state program has a somewhat different emphasis, however, each will define success somewhat differently. WELPAN members also recognize that other measures could be added, such as institutional or budgetary impacts.
WELPAN suggests that the worksheets be used by states or counties as tools to facilitate discussion about how to assess performance. The first column, listing the areas of concern and particular measures, can serve as a checklist to decide which of the areas are important and which measures should be collected. The second column should stimulate thinking about the rationale for choosing this particular measure. How is it related to the area of concern and the policy or program intervention? The third column is more technical, and refers to identifying specific sources of data that are available in the state and that can be used as a measure or proxy measure of the outcomes chosen.
|Areas of Concern and Example Measures||Rationale||Available Data|
|1. Welfare Dependency--Families use of public assistance programs to meet their basic economic needs.|
|caseload counts (one & two parent families)|
|reasons for exit|
|average grant amounts|
|use of other public services|
|average duration of welfare spell|
|use of other public services|
|2. Self-sufficiency--Welfare client's attachment to the work force.|
|type of work (paid, subsidized, community)|
|enrollment in training|
|3. Economic well-being--Family income and other sources of financial well-being.|
|4. Parental responsibility--Parents' fulfillment of their financial responsibilities toward their children.|
|paternity establishment rates|
|child support payments|
|Areas of Concern and Example Measures||Rationale||Available Data|
|health - immunization rates|
|child care - availability, quality|
|education - academic achievement, absenteeism and drop-out rates|
|juvenile justice system contacts|
|2. Adult/Parent well-being--Parent health, education, job skills.|
|health status: receiving alcohol, drug, or mental health treatment|
|enrollment in higher education/receipt degree/certificate|
|3. Family well-being--Families' ability to provide a healthy supportive and safe environment|
|medical insurance/managed care|
|child abuse and neglect (placement in out of home care)|
|housing/homelessness (shelter occupancy)|
|4. Community involvement--Communities' support of welfare reform.|
|public attitudes toward welfare|
|business/community partnerships, safety net services|
|5. Economic development--Accessibility of jobs that encourage self-sufficiency.|
|changes in low-wage labor market|
|creation of new types of support services|
|6. Family formation--(participants & general population)--Individual's ability to form and maintain stable family bonds.|
|additional, post-program-entry births|
The almost endless list of possible outcomes from welfare reform increases the importance of selecting and prioritizing the specific outcomes that are desired. Focusing on the most important desired outcomes will increase the likelihood of achieving substantive results. But different stakeholders will select or prioritize different outcomes. In this context, how can state policy officials decide which numbers to use as measures of the progress and, ultimately, the success or failure of their welfare reform efforts? WELPAN members suggest the following five steps in the decision-making process.
Step 1: Review federal compliance requirements. The first step is to review the goals and performance objectives built into the federal legislation and subsequent regulations.
Step 2: Review the state's explicit goals for welfare reform. The second step is to identify the state's program goals and choose measures that are clearly related to these goals. Some of the goals are stated explicitly in the state authorizing legislation, others in various program and mission statements. Often states have developed goals that parallel the federal TANF goals, but sometimes they have gone beyond these and added others or developed a specific interpretation of the federal goal. For example, TANF sets out as one of its purposes "promoting marriage. and encouraging the formation and maintenance of two-parent families." Several states are interpreting this goal as assuring that their laws and regulations are "neutral" about marital status.
Step 3: Consider the broader context. The third step is to acknowledge the broader context of reform. WELPAN members found that welfare reform laws and program statements typically focus on a narrow set of core outcomes. But these welfare reform initiatives should not be viewed in isolation. As administrators, WELPAN members realize they must operate within the broader framework of values, goals and principles that underlie the history of the federal welfare program, the state welfare agency and the current governor's administration. Welfare administrators also need to be sensitive to the public consensus on values.
Step 4: Connect the intervention to the outcomes. Once the broad goals are in place, the fourth step is to connect those goals to the interventions that are planned in the state program. This means specifying a series of plausible, causal links between the planned interventions and the final outcomes. The outcomes chosen need to be logically related to the intervention itself. It does no good to measure a particular outcome unless a series of stepping stones can be put in place for how it will be achieved. Developing a clear path for behavioral change through welfare reform can seem complex. Some measures represent short-term or intermediate outcomes--those things that first must be affected before the actual goals are realized. These benchmarks create stepping stones in the path to the ultimate goal. One way to approach this process is to think through the interventions as if they were part of a plot line in a story in which A is supposed to affect B in a way that affects C, and so forth. If the current narrative doesn't link actions and events together strongly enough, articulating the plot line can help reveal important flaws in logic. The example below uses this approach to show how a parent's participation in a work program can affect a child's well-being.
Step 5: Consider feasibility and cost. The final step in choosing what outcomes and impacts to measure takes into account pragmatic issues of feasibility and cost. Some outcomes are easier to measure than others. And some outcome data--such as caseload rates, entrances and exits to the caseload, job entry, etc.--are already routinely collected by welfare agencies. Other outcome data, such as marital status or educational attainment, are not routinely collected across all agencies, but would require only a little extra cost if the decision were made to do so. Other data, such as the quality of child care, are much more difficult and expensive to collect and would require some additional public or private funding.
Success in welfare reform can be defined and measured in many different ways and not all numbers are alike. To have a meaningful discussion, WELPAN members saw a need to clarify the types of measures, units of analysis and types of results that were being discussed. Thinking through how the data will be used is also important. The terms and concepts below help clarify some of the issues that are relevant to discussions of success for welfare reform.
Three different types of measures are often used interchangeably when discussing the results of welfare reforms: indicators, outcomes and impacts. These terms each mean something different and should not be confused.
There are two types of indicators. The first are measures of behavior, conditions or status that can be tracked over time, across people and/or geographic units. Examples are state marriage and divorce rates or poverty rates. Indicators typically track the behavior or situations of broad population groups. The second are thought of as process indicators, or measures of how well program operations are functioning. For example, one might develop measures of how promptly applications for assistance are disposed, or how accurately decisions about type and amount of benefits are made.
Tracking the progress of broad population indicators such as out-of-wedlock births or poverty rates is useful. But it is important to realize that changes in these indicators may be the result of many factors. It is usually very difficult to attribute them to specific welfare policy changes. For these reasons, indicators are not normally used for establishing causality.
Indicators are most useful in bringing attention to changes beyond the client population that may be indirectly a result of the program. For example, some believe that the imposition of work requirements on welfare clients may serve as a disincentive to out-of-wedlock childbearing in the population at large. Indicators will show if out-of-wedlock childbearing goes down, but too many other factors come into play to be able to make a direct connection.
Outcomes are numerical measures of behaviors or events that are generally believed to be a result of the policy or program of interest. Typically outcomes are positive and relate to the goals that the program hopes to achieve. Some actual outcomes, however, may be unanticipated, negative and undesired. In welfare reform, an example of a positive outcome would be an increase in the number of welfare recipients leaving the program and getting jobs. Data reporting on these kinds of outcomes is often available as part of program management data. However, outcome data are not collected in ways that make it possible to say that reforms have definitely caused the positive or negative change in the indicator. Absent a rigorous (often through an experiment) method for collecting and analyzing data, it may not be possible to determine if falling caseloads are due to welfare reforms or to changes in the economy.
Outcomes can also be measured in the short term. Getting a job may thus be viewed as a short-term outcome--a benchmark of progress--toward the longer-run, ultimate goal of self-sufficiency.
An outcome becomes an impact only if it is collected and analyzed in such a way that one can confidently assert that it was caused by the program or policy intervention. Researchers generally agree that proving "impact" in a scientific sense requires a rigorous evaluation design. The most reliable evaluation is an experiment with random assignment of participants to experimental groups (i.e., those who experience the program) and to control groups (i.e., those who do not). For various reasons, it is not always possible or feasible to conduct such an experiment to determine overall program impact, especially when the intervention (such as welfare reform) is complex, comprehensive and continuously evolving.
In the long run, only impacts allow one to assert with confidence that a particular policy or program actually caused the outcome that was measured. To illustrate this point, welfare caseloads in most states have declined around the same time that welfare reforms began to be put in place. These positive outcomes have been encouraging, but preliminary research suggests that it is premature to count these reductions as impacts, that is, proof of the success of the welfare reform. They also could be the result of improvements in state economic climates, which have led to greater availability of jobs and less need for welfare assistance.
Outlining broad areas of concern. For each broad area of concern, such as self-sufficiency or economic well-being, actual operational measures need to be developed. Researchers do this by first breaking down the abstraction "self-sufficiency" into more specific concepts, such as labor market attachment, which can then be measured in a number of different ways. For example, labor force attachment can be measured by counting when a client first gets a job and/or by the length of time he/she remains in the job. It is useful to develop a list of sources of data for each measure, partly because they measure different aspects of the phenomenon of interest, but also because some measures may be easier to obtain than others or are more reliable and valid measures. The figure at right depicts this concept in a simple diagram.
People and Populations of Concern. Researchers refer to the individual adult, the child, the family and the community as different units and levels of analysis because each is affected by welfare reform in a different way. The desired outcomes for each also differ. For adults the outcomes may be employment, education/training and health. For a child the outcomes may be health, school attendance and academic achievement. For families they could be reduced homelessness. For neighborhoods they could be lower crime rates, stronger business ventures, higher church membership.
Researchers also make distinctions among larger groups of people. The outcomes most typically measured are those related to welfare clients and their families, who are the direct concern in welfare reform. Within this broad population there may also be an interest in determining the differential results for sub-populations of welfare families, such as those who live in rural areas as compared with urban areas or members of different minority or disability groups. In addition, there are reasons for tracking outcomes for population groups who may be indirectly affected by welfare reform. For example, potential welfare clients (low-wage workers) could be tracked to determine entry rates or deterrent effects, and those who were former welfare recipients could be followed to ascertain changes in the rates of reentry into the system.
Systems. Several broader levels of analysis are important in any overall assessment of the results of welfare reform. Welfare reform is expected to have some ripple effects on the state's economy, especially the labor market, and may affect migration patterns between states. And it will clearly affect public budgets and the public and private institutions involved in implementing the new policies. For example, welfare reform is changing welfare agencies' internal management policies and procedures, financing and accounting systems, procurement and personnel systems and many other aspects of the organizational culture. Devolution is also affecting the relationships between state and local government, and between the public agencies and the nonprofit and private sectors, as a number of functions previously being carried out by the public agency are being privatized.
Short-Term and Long-Term Results. Policy officials' time frames are often quite short. It is important to clarify results that can be measured in the short run and others that can only be assessed in the long run. For example, it may be relatively easy to report on changes in caseload rates or job placements, but it will take much longer to assess how successful the reforms have been in terms of job retention, self-sufficiency or child well-being.
Intended and Unintended Results. One of the most problematic issues in policy making is that policies designed to do good may inadvertently cause harm. (Conversely, policies may sometimes have unintended beneficial effects.) Almost any change in policy will have some anticipated negative effects on at least some individuals, groups, or institutions. However, it is important to be alert to discover negative effects that were not anticipated and are not acceptable.
Finally, it is useful to review the several purposes of evaluation and the way that evaluation data can be used in order to justify the investment of time, effort, and resources that states will need to spend in collecting measures of success. How data is used will determine the method and approach by which it is collected. Some outcome data are used to identify and describe what is going on. Others are used to reward what works and others to understand why certain policies or programs do or do not work, and hence how they can be improved.
For good management purposes. At a minimum, good descriptive and performance data--who gets what, how they get it, when they get it and with what results-can help administrators track program progress and provide them with feedback on whether the program is being implemented as intended. (Researchers call this process evaluation.) If gathered in a timely fashion, good, qualitative management data helps lead to program and policy modifications and self-correction, if necessary.
For monitoring and goal-setting purposes. Descriptive performance data can be used to help policymakers, advocates and the media keep track of a program's progress and compare it with the program's previous record, or other states' records. Also, federal and state laws and/or administrative policy may set specific numerical goals (e.g., work participation rates, caseload reductions or child immunization rates) to be reached by certain dates. Reporting these benchmark numbers regularly helps to track the progress made toward these numerical goals.
For accountability purposes. The stakes are raised when success in meeting a goal is rewarded or failure is punished. For example, the TANF law provides federal bonuses to the five states that achieve the greatest success in reducing rates of out-of-wedlock childbearing. (The challenge is to assure that states use comparable measures to decide whether success is measured in absolute or relative terms, because states start from different baselines.) Using data for accountability purposes does not, however, depend on proving causality.
Evaluative/analytic purposes. When the purpose of data collection is to understand the causal relationship between doing something (welfare reform) and an outcome (the employment or poverty rate), the required research activity shifts to a more complex and sophisticated activity, namely causal evaluation studies. Another use of evaluative data, when combined with cost data, is to determine whether the benefits achieved by the new program or policy are worth more or less than the costs of the program (cost-benefit studies).
WELPAN members, in considering how to define success in welfare reform, generally agreed upon the following insights and recommendations:
1. When thinking through the goals of welfare reform, stakeholders should consider tracking outcomes in the following key areas: welfare dependency, labor market attachment, economic well-being, parental financial responsibility, child, parent and family well-being, family formation, community involvement and economic development.
2. State officials and citizens interested in welfare reform need to carefully define their goals and measures of success. Reform is a more likely to be effective when it is clear what one wants to achieve.
3. The mandates of the federal law are only a starting point and are too narrow. States and communities should engage in a deliberative process of study and debate designed to view these goals in a broader context, to deepen understanding of the complexities involved, and to ensure broad, sustained support for the reforms.
4. Engaging in a useful and constructive dialogue about what constitutes success in welfare reform requires a common language. This means taking the time to develop agreements on key concepts, terms and definitions.
5. While recognizing that each state will define success in its own terms, WELPAN members recommend thinking about two broad types of outcomes: First, those that are of most immediate concern to policymakers and the public--such as placing welfare recipients in jobs and reducing caseloads. Second, outcomes that are longer term and ultimately the most important--such as improvements in child well-being, safer communities, more stable families.
6. Choosing outcomes and the appropriate social and economic measures to monitor them is only the first step. States must invest in sufficient resources to ensure that the right data are available and the measures are used and interpreted objectively.
Defining success in welfare reform is complex. It requires stepping back from day-to-day activities and synthesizing the various directives that are coming into welfare agencies from the federal government, state government and other stakeholders. It requires developing a common language for defining appropriate measures. It requires making clear the linkages between interventions and the desired goals.
Defining success and how to measure it is also critical to the welfare reform effort. Clear, appropriate goals will enable welfare officials to focus resources on achieving substantive results. Specific and measurable outcomes will promote effective programs and policies. The action of thinking through how interventions are connected to results will help identify weaknesses in theory and application, thus strengthening reform efforts. If articulating goals and action plans is not possible, it may be that the reform was not well-developed in the first place.
WELPAN members found this process of deliberating on the goals of welfare reform so helpful to their own efforts that they wrote this report in the hope it will provide useful tools to others who are trying to achieve the multiple goals of welfare reform.