This article is also available in PDF File.

The 1994 Amendments to the Social Security Act authorized the U.S. Department of Health and Human Ser-vices to review State child and family service programs in order to ensure substantial conformity with the State plan requirements in titles IV-B and IV-E of the Social Security Act (the Act). Federal regulations at 45 Code of Federal Regulation (CFR) 1357.15 implement the requirements for the States' plan as the child and family services plans (CFSP), and to submit the CFSP to the Administration for Children and Families (ACF). In the event that ACF determines a State is not in substantial conformity with any one of the outcomes or systemic factors, a program improvement plan is required. (See 45 CFR 1355.35.) The State must address in its program improvement plan each outcome/systemic factor that was determined not to be in substantial conformity.

The goal of this publication from the National Resource Center for Child Welfare Data and Technology (NRC-CWDT) is to assist States on how benchmarks are used in the program improvement plan. Benchmarks are a way to review and capture the experience of child welfare as States work towards clear and specific goals.

What are Benchmarks?
For purposes of the Child and Family Services Review (CFSR) program improvement plans (PIP) benchmarks are measurable indicators of progress occurring at established intervals in a strategy to achieve a goal. The type of bench-marks the State establishes in its PIP will depend upon the strategies the State uses and the duration of the strategies.

What are the different kinds of Benchmarks?
Benchmarks can be either quantitative indicators of progress or they can be more process-oriented. Regardless of the type of benchmarks the State uses, they must be clearly measurable, specific in their definition (i.e. specify the de-sired results and the specific timeframe), and should accurately reflect the strategy they are designed to measure.

Quantitative benchmarks are usually appropriate when the State establishes a goal of achieving a numeric change in performance in some area. For example, a State may have an overall goal of achieving a 5% increase in the number of children who are reunified with their parents within 12 months of entering foster care over a two year PIP. Depending on what actions the State decides to take to reach that goal, and how long it will take those actions to have an effect on the rate of reuni-fication, the State may decide to project bench-marks that include a one percent increase after the first three quarters of implementing the strategy, a 2.5% increase after 15 months, a 4% increase at 18 months, leading to the full 5% increase at the conclusion of the PIP.

Process-oriented benchmarks are usually appropriate where the actual goal in the PIP is to implement new procedures. For example, one of the State's goals may be to train all child welfare workers in risk management. Implementation strategy, the State may identify all the action steps required to achieve that goal and the time frames in which it will take those action steps.

States may also use a combination of quantitative and process-oriented benchmarks within the same strategies. Using the examples above, the State can establish all the action steps and time frames needed to train staff in risk assessment, and simultaneously establish quantitative goals and benchmarks that measure the effectiveness of the training, i.e., increases in the number of children who are reunified with their parents.

Why use Benchmarks?
Benchmarks are especially important for long-term strategies to improve outcomes and services. Benchmarks allow the State to gauge its progress along the way and, if necessary, to modify its strategies and plans within the time frames of its PIP in order to achieve the desired results.

Benchmarks are used to:

• Help States to determine realistic goals for program improvement.
  
• Allow States to evaluate or assess their progress periodically.
  
• Evaluate progress at strategic intervals, rather than just at the end of the PIP.
  
• Modify indicated strategies to ensure the goals and objectives of the PIP are met.
  
• Report progress on the PIP to State and Federal partners in the CFSR process.

How can a state use Benchmarks to evaluate is PIP strategy?
The following example illustrates how a State might use combined benchmarks to evaluate and modify a strategy in its PIP:

A State establishes a goal to reduce the number of foster care re-entries. An activity to reach the goal could be to provide a certain type of training to all child welfare staff by December 2003. The State should first consider factors such as whether the date is realistic when a curriculum needs to be developed and training sites identified, whether equipment is in place, whether training staff are available, and whether there are issues outside the control of the agency that could have an impact such as budget allocations, contract agreements, or delivery of equipment? Considering these factors will help the State determine the interim dates by which it will achieve each step toward the ultimate goal.

One benchmark may be to have the curriculum developed by outside contractors by a certain date in 2002. However, due to issues outside the control of the State agency the contract for the curriculum may get delayed and the deliverable may be six months late. When the State evaluates the progress, recognizes they are six months behind schedule, and compares the effect of not meeting this benchmark on subsequent benchmarks, the State may determine that it needs to work with the Regional Office to modify the strategy. It may, for example, decide that in order to reach its goal within the time frame for the PIP, it needs to train more people in less time by increasing the number of trainers and training sessions.

By establishing benchmarks and corresponding evaluation criteria in the PIP, the State is able to evaluate progress, identify barriers, modify the schedule, and proceed to reach the goal on time.

Without having established benchmarks and evaluating them periodically, or at least quarterly, the State takes a risk of being unsuccessful in attaining the goals in its PIP and incurring penalties for failing to achieve its goals, but more importantly, risks being unsuccessful in attaining the goals as outlined in the PIP, or completing initiatives that were not goals of the PIP.

Where do states record Benchmarks in their Program Improvement Plans (PIP)?
Information Memorandum ACYF-CB-IM-02-04 provides guidance and assistance to States in developing PIPs in response to the findings of the Child and Family Service reviews, and a suggested format for preparing the PIP for submission to the ACF Regional Office. The regulation, at 45 CFR 1355.35(a), requires States to include certain information in the PIP, although no specific format for developing the PIP is prescribed. Further, at 45 CFR 1355.35(d)(4), States are required to submit quarterly status reports to the Administration for Children and Families (ACF), unless less frequent reports are mutually agreed upon, to inform ACF of the State's progress in implementing the provisions of the PIP. Measurable benchmarks are a required component of the PIP and clearly have a pivotal function in the PIP matrix described in AFYC-CB-IM-02-04. Specific references to benchmarks include:

• Column 4 of the PIP Matrix, States summarize their evaluation approach to measuring PIP benchmarks and goals.
• Column 5 of the PIP Matrix, States list the quarterly benchmarks for conducting in terim monitoring of PIP progress for each applicable item and systemic factor. Benchmarks are the interim and measurable indicators that will be used to determine if progress is being made toward achieving the established goal.
• Column 6 of the PIP Matrix, States include the projected dates for achieving the quarterly benchmarks quarterly benchmarks listed in column 5. In each quarterly report (using the PIP Matrix), States note the benchmarks that have been achieved.
• PIP Narrative Report Form: When benchmarks or goals are not met, States should provide a narrative explanation in this section.

Where else in the PIP is reference made to Benchmarks?
On the PIP Quality Assurance Checklist, there is an indicator "Does the plan include benchmarks for measuring improvements?" and another one for "Are those benchmarks consistent with the level of effort required to improve per-formance?"

What happens if a state does not meet the Benchmarks in its PIP?
When a State is not in substantial conformity with the outcomes and systemic factors reviewed in a CFSR, it is subject to financial pen-alties based on the non-conformity. However, if the State succeeds in implementing the provisions of a Federally approved PIP that addresses the non-conformity, the potential penalties may be rescinded by the Administration for Children and Families (ACF) at the time during the PIP when the goal is achieved. If however, the State does not achieve the goals in its PIP, the assessed penalties may be withheld from Federal funds due to the State. If the State does not meet critical benchmarks identified in the PIP, rather than waiting until the end of the PIP, ACF may begin to withhold Federal funds at the time when the goal was to be achieved. It is, therefore, important that the State carefully consider the time frames needed to achieve it goals and reach its benchmarks, and to use benchmarks to evaluate its progress and make adjustments as needed.

This document was prepared as a supplement to the material presented in a session at the April 2002 Making IT Work: Using Data for Program Improvement in Arlington, VA. We hope that it will serve to stimulate an exchange of ideas and information among States and between systems and program staff. Your feedback is important to us. If you have any additional information on the topic presented in this sheet, or if you have any comments or suggestions regarding its presentation or content, please contact Elaine Vocesof Xtria at (703) 821-3090 x241 or nrccwdt@xtria.com