Performance audit, department of economic security, division of children, youth and families, child protective services, in home services program

              Debra K. Davenport
Auditor General
Performance Audit
Department of
Economic Security—
Division of Children, Youth and Families—Child
Protective Services—In-Home Services Program
Performance Audit Division
April • 2012
REPORT NO. CPS-1201
A REPORT
TO THE
ARIZONA LEGISLATURE
The Auditor General is appointed by the Joint Legislative Audit Committee, a bipartisan committee composed of five senators
and five representatives. Her mission is to provide independent and impartial information and specific recommendations to im-prove
the operations of state and local government entities. To this end, she provides financial audits and accounting services to
the State and political subdivisions, investigates possible misuse of public monies, and conducts performance audits of school
districts, state agencies, and the programs they administer.
The Joint Legislative Audit Committee
Audit Staff
Copies of the Auditor General’s reports are free.
You may request them by contacting us at:
Office of the Auditor General
2910 N. 44th Street, Suite 410 • Phoenix, AZ 85018 • (602) 553-0333
Additionally, many of our reports can be found in electronic format at:
www.azauditor.gov
Dale Chapman, Director
Catherine Dahlquist, Manager and Contact Person
Daniel Hunt
Representative Carl Seel, Chair
Representative Tom Chabin
Representative Justin Olson
Representative David Stevens
Representative Anna Tovar
Representative Andy Tobin (ex officio)
Senator Rick Murphy, Vice Chair
Senator Andy Biggs
Senator Rich Crandall
Senator Linda Lopez
Senator David Lujan
Senator Steve Pierce (ex officio)
2910 NORTH 44th STREET • SUITE 410 • PHOENIX, ARIZONA 85018 • (602) 553 -0333 • FAX (602) 553-0051
MELANIE M. CHESNEY
DEPUTY AUDITOR GENERAL
DEBRA K. DAVENPORT, CPA
AUDITOR GENERAL
STATE OF ARIZONA
OFFICE OF THE
AUDITOR GENERAL
April 3, 2012
Members of the Arizona Legislature
The Honorable Janice K. Brewer, Governor
Mr. Clarence Carter, Director
Department of Economic Security
Transmitted herewith is a report of the Auditor General, a performance audit of the
Department of Economic Security, Division of Children, Youth and Families—Child
Protective Services—In-Home Services Program. This report was prepared pursuant to and
under the authority vested in the Auditor General by Arizona Revised Statutes §41-1966.
As outlined in its response, the Department of Economic Security agrees with the findings
and plans to implement or implement in a different manner all of the recommendations.
My staff and I will be pleased to discuss or clarify items in the report.
This report will be released to the public on April 4, 2012.
Sincerely,
Debbie Davenport
Auditor General
Attachment
In-Home Services Program
The Division of Children,
Youth and Families (Division)
can strengthen its in-home
services program
by taking steps to further
support in-home services
contractors’ use of evidence-based
practices, which are
interventions, programs, or
treatments that have been
established as effective
through scientific research.
Despite providing a wide
array of services to help
families, the impact of
the Division’s in-home
services program is mixed.
Literature indicates that
providing services that are
evidence-based may yield
better results. Therefore,
the Division should
take additional steps to
incorporate these practices
in its in-home services
program. These steps
include communicating
its intent that services be
evidence-based and making
it a requirement in the next
in-home services contract
solicitation, developing well-defined
criteria for identifying
evidence-based practices,
maintaining an updated
listing of these practices, and
monitoring its contractors
to ensure that they are
implementing evidence-based
practices as designed.
The Division's in-home services program provides voluntary, time-limited
services—up to 120 days—to help stabilize, strengthen, and preserve
families. In-home services are offered to families with unresolved
problems or a home situation that presents actual or potential risk to a
child’s safety or well-being. These services seek to alleviate risks so that
children can remain safely at home or be reunified with their families.
The Division’s in-home services program provides a continuum of family-centered
services. These services include crisis intervention counseling,
conflict resolution and anger management skills development, and job
readiness training. Additionally, the in-home services program assists
families to access services such as substance abuse treatment, housing,
and child care; and transition children to less restrictive placements, for
example, from a foster home to a family home. Although the same array
of services is available for all families, the specific services provided to a
family and their frequency and duration is tailored to the family’s
individual risks and needs.
Most in-home services are provided to families by contractors, with
division in-home services staff providing case management and
assessment services. The contractors use in-home services teams
composed of a team lead and a family support worker to provide
services to families. In-home services are provided primarily during visits
with families in their home, at the location of the child’s current placement,
or at Child Protective Services (CPS) offices. In fiscal year 2011, the
Division spent more than $11.4 million for contracted in-home services.
Additionally, the Division spent an estimated $8.8 million on salary and
employee-related expenses for division in-home services staff.
Division can strengthen in-home services
program by continuing to move toward the use
of evidence-based practices (see pages 7
through 19)
The Division can strengthen its in-home services program by further
supporting in-home services contractors’ use of evidence-based
practices. Evidence-based practices are interventions, programs, or
treatments that have been established as effective through scientific
research. Despite providing a wide array of services to help preserve,
page i
Office of the Auditor General
SUMMARY
This audit was conducted
under the authority vested
in the Auditor General by
Arizona Revised Statutes
§41-1966.
Agency Comments
The Department of
Economic Security
(Department) agrees
with the findings and will
implement or implement in
a different manner all of the
recommendations.
Our Conclusion
page ii
State of Arizona
support, and reunify/stabilize families at risk for child abuse and neglect, the impact
of the Division’s in-home services program on keeping children safely in their homes
and improving family functioning is mixed. Literature indicates that providing services
that are evidence-based may yield better results. For example, parent-child
interaction therapy is an evidence-based practice that has been shown to reduce the
recurrence of physical abuse of young children with emotional and behavioral
problems, whereas Brief Strategic Family Therapy® has been shown to improve
family functioning. The Division redesigned the in-home services contracts to require
contractors to have evidence-based practices available for several types of in-home
services. However, these contracts, which were awarded in May 2011, do not
actually require contractors to use evidence-based practices.1 Therefore, the Division
should take the following additional steps:
• The Division should communicate with its stakeholders, including existing and
potential contractors, its intent that future provision of in-home services must be
evidence-based and then make this a requirement in the next contract
solicitation. This would allow existing and potential contractors time to develop
or expand their capacity to implement evidence-based practices for in-home
services.
• The Division should develop well-defined, written criteria to identify appropriate
evidence-based practices for the in-home services program and maintain an
updated inventory of these practices. This information should be made available
through the Department’s Web site. Although the Division’s in-home services
contracts define evidence-based practices, they do not include clear criteria for
identifying these practices, which may result in a wide variation in the quality and
effectiveness of the services provided. The Division should work with its in-home
services contractors and other knowledgeable sources, such as the National
Resource Center for In-Home Services, a national center of expertise regarding
child welfare practice, to develop the criteria.
• Finally, while its contractors begin to incorporate evidence-based practices into
the services provided to division clients, the Division should expand its
monitoring of its contractors to ensure the contractors are implementing
evidence-based practices as designed. Ensuring that a service or intervention
is provided as designed involves obtaining information on how closely the
implementation adheres to the practice’s essential components, including
staffing, training, content, and program delivery. Further, the Division should
require contractors that modify evidence-based practices to provide written
1 Although the Department released the in-home services contract solicitation in December 2010, in accordance with
Arizona Administrative Code R2-7-A902, it postponed implementation of the contracts soon after awarding them in May
2011 so that the Department could review and render a decision on several bidder protests regarding the procurement
process. Four entities appealed the Department’s decisions to the Department of Administration (DOA). As of March
1, 2012, one entity’s protest was still pending a decision by DOA. Further, another entity requested a rehearing of DOA’s
decision. To ensure that in-home services were not disrupted as a result of the postponed contracts, the Department
extended the existing in-home services contracts that would have terminated in May 2011 pending final resolution of
the protests.
page iii
Office of the Auditor General
justification for the modifications to verify that essential components are not
being modified without the approval of the practices’ developers.
Division’s in-home services program
intended to stabilize, strengthen, and
preserve families
The Division of Children,
Youth and Families’
(Division) in-home services
program provides a wide
array of services to help
families. These services
include crisis intervention
counseling and anger
management. The in-home
services program also
helps families access
services such as substance
abuse treatment and child
care. Services may also be
provided to transition a
child from a more
restrictive residential
placement back to a family
home. Although the same
array of services is
available for all families, the
specific services provided
and their frequency and
duration is tailored to the
family’s individual risks and
needs. Most in-home
services are delivered
through contractors, with
division staff providing case
management and
assessment services to the
families. In-home services
are provided primarily
during visits with families in
their home, at the location
of the child’s current
placement, or at Child
Protective Services’ (CPS)
offices. In fiscal year 2011,
the Division spent more
than $11.4 million for
contracted in-home
services. In addition, the
Division spent an estimated
$8.8 million on salary and
employee-related
expenses for division
in-home services staff.
The Division’s in-home services program provides voluntary, time-limited
services—up to 120 days—to help stabilize, strengthen, and preserve
families. In-home services are offered to families with unresolved problems
or a home situation that presents actual or potential risk to a child’s safety
or well-being. These services seek to alleviate risks so that children can
remain safely at home or be reunified with their families.
In-home services program provides a variety of services to
at-risk families—The Division’s in-home services program provides
a continuum of family-centered services that are comprehensive,
coordinated, community-based, accessible, and culturally responsive.1
These services include crisis intervention counseling, conflict resolution
and anger management skills development, and job readiness training
(see textbox on page 2 for other available services). The in-home
services program also assists families to access services such as
substance abuse treatment, housing, and child care. Services may also
be provided to transition a child from a more restrictive residential
placement back to a foster or family home, or from a foster home to a
family home.
Although the same array of services is available for all families, the
specific services provided to a family and their frequency and duration
is tailored to the family’s individual risks and needs. For example, one
family was reported to CPS because the father and his teenage son
were not getting along and there were allegations of physical abuse. The
family was provided with 34 hours of direct services from the in-home
services team over a 3-month period. The services provided were parent
skills training, communication skills development, and anger
management. Another family consisting of teenage parents with two
children, one born substance-exposed, was provided with approximately
32 hours of direct services from the in-home services team over a
4-month period. During that time, the family received counseling and
parenting skills training, information on substance abuse and community
1 The Division’s in-home services program incorporates characteristics of family preservation models. Since
the 1970s, a number of family preservation programs have been developed to provide services to children
and families experiencing serious problems that may lead to the placement of children in foster care or
otherwise result in the dissolution of the family unit. Although the programs shared a common philosophy
of family-centered services, meaning that services focus on the entire family rather than select individuals
within a family, they differed in their treatment theory, level of intensity of services, and length of service
provision.
page 1
Office of the Auditor General
Background
page 2
State of Arizona
supports, bus tickets, and emergency funds for children’s clothing. In addition,
CPS referred both parents for substance abuse services. The case was eventually
closed because the family was unwilling to continue with services. The in-home
services team reported at the time of case closure that there were no current safety
concerns because the children were residing with their maternal grandparents,
who were appropriate caregivers.
Contractors primarily provide in-home services—Although division staff
provide case management and assessment services to families in the in-home
services program, most in-home services are delivered through contractors.
These contractors use in-home services teams composed of a team lead and a
family support worker to provide services to families.1,2 During the initial meeting
with the family, contract staff will develop an initial interim plan that outlines the
expectations of the family and contractor for the next 30 days. During this initial
30-day period, contract staff will conduct a comprehensive assessment of the
1 The team lead is a master's-level professional or a bachelor's-level professional with 5 years of work-related experience.
The family support worker is a bachelor's-level professional or a paraprofessional with 5 years of work-related
experience.
2 Some contracted services for the support and preservation of families were also provided through a division program
known as Arizona Promoting Safe and Stable Families (APSSF). Although similar services were provided, they were
delivered differently. For example, the APSSF program was not required to use in-home services teams or conduct a
comprehensive assessment. APSSF services were suspended in March 2009 for nontribal contracts and April 2009 for
tribal contracts due to division budget reductions. The Division has since redirected the resources from this program
as they became available to expand the in-home services program.
In-home services provided
• Family assessment
• Goal setting and case planning
• Parent education and skills training
• Individual, marital, and family therapy
• Domestic violence treatment and/or education
• Behavior management and modification
• Nutrition and home management education
• Linkages to community resources
• Skills development in:
• Communication and negotiation
• Problem solving and stress management
• Emergency funds to purchase needed items/resources not
otherwise available and deemed essential to family functioning.
These funds may not exceed a total of $300 per family.
Source: The Department of Economic Security’s (Department) in-home services
contract solicitation released in December 2010.
page 3
Office of the Auditor General
family and develop a service plan based upon the family’s risks and needs.
In-home services are provided primarily during visits with families in their home, at
the location of the child’s current placement, or at CPS offices.1 As shown in Table
1 (see page 4), the frequency of home visits and their cost will vary depending on
the purpose of the visits and services, i.e., family preservation, support, and/or
reunification and stabilization.
The Division provided in-home services to nearly 75 percent of approximately
59,000 children associated with CPS reports in Arizona in federal fiscal year
2010.2,3 The Division uses State General Fund and federal monies to pay for
in-home services. As shown in Table 2 (see page 5), in fiscal year 2011, the
Division spent more than $11.4 million for contracted in-home services to support,
preserve, and reunify/stabilize families. State General Fund monies represented 91
percent of this amount. In addition, the Division spent an estimated $8.8 million on
salary- and employee-related expenses for division in-home services staff who
provide case management and additional support, including conducting monthly
visits with the families and completing required safety and risk assessments.
1 According to division management, although it is the intent that in-home services primarily be home-based, the
program does not exclude services being offered in the contractors’ facilities. Additionally, the in-home services
contracts require that families be connected to community supports. Depending on the individual's assessed needs, a
family member may be referred to a community group, such as Alcoholics Anonymous, as part of his/her service plan.
2 U.S. Department of Health and Human Services, 2011
3 The reported percentage is based on children who received only in-home services. An additional 11 percent of the
children who were associated with CPS reports received foster care services and may have also received in-home
services.
page 4
State of Arizona
Table 1: Minimum Required Frequency of Home Visits and Average Cost
By Service Level1
1 The time frames and costs are based on the contracts the Department awarded in May 2011. As of March 1, 2012, these contracts
had not been implemented because of several bidder protests regarding the procurement process. As of March 1, 2012, two protests
were still under review by the Department of Administration.
2 The same array of services are available for all levels and are provided primarily during visits with the family in their home, the child’s
current placement, or CPS offices.
3 The Division may grant exceptions to the number of visits by an in-home services team based on a family’s needs.
4 To make comparisons across time frames and service levels, one time per week was converted to four times per month. Similarly, two
times per week was converted to eight times per month.
5 Services to support families can be provided for up to 120 days, but can also be provided for a shorter period.
Source: The Department’s in-home services contract solicitation released in December 2010.
Frequency of Home Visits3,4
(Team Lead / Family Support Worker)
Service Level2
1st
Month
2nd
Month
3rd
Month
4th
Month
Intensive Family Preservation—Provides crisis-oriented
services to families where conditions present a threat to
child safety and the children are at significant risk of out-of-home
placement due to abuse and/or neglect.
Average payment rate is approximately $4,700 per case.
4 / 8
4 / 8
2 / 8
1 / 4
Moderate Family Preservation—Provides services to
families where conditions do not present a safety threat to
the children, but a high to moderate risk of abuse and/or
neglect exists.
Average payment rate is approximately $3,400 per case.
4 / 4
4 / 4
2 / 4
NA
Family Support5—Provides short-term support to families
where conditions present a potential or low risk of abuse
and/or neglect to the children. This service may be
provided to families referred by CPS, community families,
or self-referrals.
Average payment rate is approximately $70 per hour.
2 / 4
2 / 4
2 / 4
2 / 4
Family Reunification and Stabilization—Provides
services to safely expedite the return of children who are in
out-of-home placement or in voluntary foster care to their
family, or to transition a child from a more restrictive
placement back to the community. These services may
also be used to assist in stabilizing or safely maintaining a
child in a relative/kinship or adoptive home.
Average payment rate is approximately $3,800 per case.
4 / 4
4 / 4
2 / 4
1 / 2
page 5
Office of the Auditor General
1 In March 2009, the Division reduced in-home services expenditures in response to budget reductions and shortfalls. However, in fiscal year 2010, the
Division again encouraged staff to serve families in their homes by developing safety plans to control safety threats while providing contracted or
community in-home services.
2 Some contracted services for the support and preservation of families were also provided through the APSSF program. APSSF services were suspended
in March 2009 for nontribal contracts and April 2009 for tribal contracts due to division budget reductions. The Division has since redirected the resources
from this program as they became available to expand the in-home services program.
Source: Auditor General staff analysis of division expenditure data for state fiscal years 2009 through 2011 maintained on the Children’s Information
Library and Data Source (CHILDS) system.
Table 2: Expenditures for Contracted In-home Services
Fiscal Years 2009 through 20111,2
2009 2010 2011
Amount Percent Amount Percent Amount Percent
Federal $13,119,000 67 $ 781,000 9 $ 1,029,000 9
State 5,780,000 29 7,928,000 91 10,435,000 91
Other 750,000 4 0 0 0 0
Total $19,649,000 100 $8,709,000 100 $11,464,000 100
page 6
State of Arizona
Division can strengthen in-home services
program by continuing to move toward the
use of evidence-based practices
The Division of Children,
Youth and Families
(Division) can strengthen its
in-home services program
by taking steps to further
support in-home services
contractors’ use of
evidence-based practices.
These practices are
interventions, programs, or
treatments that have been
established as effective
through scientific research.
Despite providing a wide
array of services to help
families, the impact of the
Division’s in-home services
program is mixed.
Literature indicates that
providing services that are
evidence-based yields
better results. Division
in-home services contracts
awarded in May 2011
require that contractors
have these practices
available, but do not
actually require contractors
to use evidence-based
practices. The Division
should take additional
steps to incorporate these
practices into its in-home
services program. These
steps include
communicating its intent
that in the future, in-home
services be evidence-based
and making it a
requirement in the next
in-home services contract
solicitation, developing
well-defined criteria for
identifying evidence-based
practices, maintaining an
updated listing of these
practices, and monitoring
its contractors to ensure
that they are implementing
evidence-based practices
as designed.
Impact of Division’s in-home services is mixed
Despite providing a wide array of services to help preserve, support, and
reunify/stabilize families at risk for child abuse and neglect, the Division’s
in-home services program has produced mixed results on keeping children
safely in their homes and improving family functioning. Specifically:
• In-home services provided to help preserve families produce
mixed results—The Division provides in-home services to families in
crisis with the goal of protecting the child, strengthening the family,
and preventing unnecessary out-of-home placement. The Division
seeks to meet this goal by stabilizing the family, i.e., helping the family
ensure all members are safe, giving the family the tools needed to
care for and protect their children, improving family functioning, and
building connections to support networks in the community. The
services, including crisis intervention, individual and family counseling,
and parent training, share the same characteristics as other family-centered
services and are community-based. However, these services
are delivered in a more intensive and targeted manner, focus on
families at greater risk of having a child placed in out-of-home care,
and are short-term. These types of preservation services are most
often provided to families who have come to the attention of the child
welfare, mental health, or juvenile justice systems because of child
abuse or neglect, child behavioral health challenges, delinquency, or
serious parent-child conflict.
However, auditors’ review of 14 cases where families received
intensive in-home services to help preserve their families found mixed
results.1 Specifically, 7 of these families received Child Protective
Services (CPS) reports after receiving services, with 6 reports resulting
in child removals (see textbox—case example 1 on page 8). Another
family who did not receive a subsequent CPS report voluntarily placed
its child with a relative to prevent CPS from filing a dependency
1 Two of the intensive in-home services cases were closed because the contracted in-home services teams
were unable to contact the families. In one case, the family was reassigned to another in-home services
contractor, but this case was also closed because the child was going to receive services through the
Juvenile Probation Office. In the other case, moderate in-home services were opened for the family 2
months later. Neither of these families received subsequent CPS reports or had children placed in out-of-home
care.
page 7
Office of the Auditor General
FINDING 1
petition and taking the child into state custody. The families who did not have
children removed from home generally showed modest improvement/change
in family functioning (see textbox—case example 2).
Auditors found similar results for the 19 case files reviewed for families who
received moderate in-home services to help preserve their families.1 After
receiving services, 5 of the families had children placed in out-of-home care.
According to case documentation, only 7 of the 19 families completed services,
and several of these families continued to struggle with the issues that resulted
in their initially receiving services. For example, one family could not adequately
care for and control one of its children, so the child was placed in the care of an
adult sister who was provided with financial assistance and help with other basic
needs such as food. Although the adult sister provided a stable environment for
her younger sibling, the child continued to run away and eventually ended up in
juvenile detention for violating probation. Another family was referred for services
1 Two of the moderate in-home services cases were closed because the contracted in-home services teams were unable
to contact the families. There were no subsequent CPS reports on either family, nor were any children placed in out-of-home
care.
page 8
State of Arizona
Case example 1
In August 2008, CPS referred a family for intensive in-home services to help preserve the family after the
mother gave birth to her fourth substance-exposed infant. The Division provided 3 months of intensive services
to this family, including 21 hours of face-to-face contact with the family from the contracted in-home services
staff during which the family received parenting education and grief counseling. The family also received $300
in emergency funds. After the family received services, the Division approved the case for closure because the
mother had completed most of her goals and the children were determined to be safe. Approximately 3
months later, the family received another CPS report involving the parents’ abuse of substances and inability to
care for the children. The children were adjudicated dependent by the courts and placed in the care of
relatives. The parents’ rights were terminated a year later. In addition to the services provided in 2008, the
family had also received substance abuse treatment, parent aide assistance, and moderate in-home services
in 2007, and intensive in-home services in 2002 and 2006.
Case example 2
In September 2008, CPS referred a family for intensive in-home services to help preserve the family because
the mother could not control her 12- and 13-year-old children who would run away, treat the mother
disrespectfully, and did not attend counseling for their behavioral issues. After 4 months of intensive services,
including 26 hours of face-to-face contact with the family from the contracted in-home services staff during
which the mother received parenting skills training and help obtaining public assistance, the family had only
marginally improved. Although the family was receiving healthcare, food stamps, and cash assistance,
according to case documents, the mother could not consistently implement the parenting strategies she was
taught, and her children had the same issues. In addition, the family struggled to maintain stable housing.
Eight months after receiving services, the family received another CPS report and was still experiencing similar
problems.
Source: Auditor General staff summary of two division in-home services cases.
because of family stress and the father’s alcohol use. After completing services,
which included parent education on stages of child development and
nonphysical forms of discipline, case documentation noted that these risks
remained and identified additional risks related to the parents’ ability to manage
the child and child’s behavior. Four months after the family received services,
another CPS report was made, and the family was then referred for intensive
in-home services to help preserve the family.
A 2009 evaluation of the Division’s intensive in-home services also found mixed
impact.1 This evaluation examined whether families felt they were stronger after
participating in services. The evaluation found that 39 of the 53 families—74
percent—participating in the evaluation reported that they felt their families were
getting stronger. These families reported improved parenting skills, improved
communication and relationship skills, and progress with addiction/substance
abuse issues. However, only 32 of the 53 families—60 percent—attributed
some of the positive change to their involvement in in-home services. Further,
measures of family functioning that evaluators administered prior to beginning
services and after services found that 14 of the 30 families—47 percent—who
participated in this assessment showed virtually the same or lower level of family
functioning after receiving services. The literature on the effectiveness of similar
types of services to preserve families also reports mixed results.2
• Evaluations of the Division’s in-home services to help support families
reported success, but literature on the effectiveness of these types of
services shows more mixed results—In-home services aimed at supporting
families are intended to help parents provide stable and nurturing homes,
promote safe environments, and enable healthy child development. Examples
of support services include education and training to promote parents’ skills and
understanding of child development, discipline, and communication; and job
training to develop specific vocational skills. Similar to preservation services,
supportive services are community based, family focused, and short term.
However, they are generally less intensive and may be provided to families with
potential or low risk of abuse and/or neglect referred through CPS, community
agencies such as law enforcement, or self-referral.
Although evaluations of the Division’s family support services have reported
success in addressing families’ needs and preventing child abuse and neglect,
some of the results should be interpreted with care. For example, a 2008
evaluation reported that most families who received parent skills or parent aide
1 Lietz, 2009
2 Berry, Propp, & Martens, 2007; Biehal, 2005; Blythe & Jayaratne, 2002; Cash & Berry, 2003; Chaffin, Bonner, & Hill,
2001; Dagenais, Begin, Bouchard, & Fortin, 2004; Heneghan, Horwitz, & Leventhal, 1996; Khan, Moore, & Moore,
2010; Kirk & Griffith, 2006; Kirk & Griffith, 2004; Lindsey, n.d.; Lindsey, Martin, & Doh, 2002; Littell & Schuerman, 2002;
Littell & Schuerman, 1995; MacLeod, & Nelson, 2000; McCroskey & Meezan, 1998; Miller, 2006; Nelson, Walters,
Schweitzer, Blyth, & Pecora, 2009; O’Reilly, Wilkes, Luck, & Jackson, 2010; Pope, Williams, Sirles, & Lally, 2005; Roberts
& Everly, 2006; Ryan & Schuerman, 2004; Tully, 2008; Tyuse, Hong, & Stretch, 2010; and Westat, James Bell Associates,
& Chapin Hall, 2002
page 9
Office of the Auditor General
page 10
State of Arizona
training reported improved parental competence.1 However, this result must be
interpreted with caution because the survey results were based on a convenience
sample comprising primarily families who completed services rather than a
random sample of all participating families, which would have provided a more
complete, truer picture of the impact these services had on participating
families. The 2008 evaluation also reported very low rates of substantiated child
maltreatment within 6 months of program participation, which would appear to
be a positive outcome of the Division’s family support services. However, as
shown in Table 3, the low rate of substantiated child maltreatment may also
reflect the State’s historically low rates for substantiated child maltreatment.
Literature on the effectiveness of family support services reports more mixed
results.2 For example, a national evaluation examined the impact of 260 family
support programs on selected child and adult outcomes such as parenting
behavior, child safety, and family functioning and found that these programs had
small but significant effects.3,4 As a result, the authors cautioned against making
strong claims for family support services as an intervention strategy likely to
make meaningful differences in families’ lives. The evaluation found that
1 LeCroy & Milligan Associates, 2008
2 Chaffin et al., 2001; Duggan, Caldera, Rodriguez, Burrell, Rhode, & Crowne, 2007; Howard & Brooks-Gunn, 2009;
Howing, Wodarski, Gaudin, & Kurtz, 1989; Kahn et al., 2010; Layzer, Goodson, Bernstein, & Price, 2001; LeCroy &
Krysik, 2011; MacLeod & Nelson, 2000; O'Reilly et al., 2010; Pope et al., 2005; Reynolds, Mathieson, & Topitzes, 2009;
and Waldfogel, 2009
3 Layzer et al., 2001
4 The convention in the social sciences is that effect sizes below 0.2 are not educationally meaningful. Effect sizes
between 0.2 and 0.5 are considered small and potentially meaningful, and only effect sizes larger than 0.8 are
considered large. In three areas—child cognitive achievement, child social and emotional functioning, and parenting
behavior—the effects were consistently meaningful, albeit small.
Source: Auditor General staff analysis of the U.S. Department of Health and Human Services
annual Child Maltreatment reports for federal fiscal years 2005 through 2010.
Table 3: Comparison of Arizona and National Percentages
of Substantiated CPS Reports
Fiscal Years 2005 through 2010
Fiscal
Year Arizona National
2005 12% 28%
2006 9% 29%
2007 9% 25%
2008 9% 24%
2009 9% 24%
2010 16% 26%
page 11
Office of the Auditor General
although the core service provided is some form of parenting education, almost
two-thirds of the programs studied had very small or no effects on parents’
understanding of child development, attitudes about childrearing, or behavior
with their children. Further, more than half of the programs evaluated had small
or no effects on family functioning. Additionally, a 2001 study that examined
child maltreatment outcomes across client risk levels and program types among
an entire state-wide group of family preservation and family support programs
found that families completing program services did not differ from those who
dropped out or received only one-time services, and there was no relationship
between program intensity or duration and outcomes.1
• Division’s in-home services also produced mixed results at helping reunify/
stabilize families—Reunification and stabilization services aim to safely
expedite the return of children who are in out-of-home care back to their family,
transition a child from a more restrictive placement back to the community, or
assist in stabilizing or safely maintaining a child in a relative or adoptive home.
These services include individual and family counseling, anger and stress
management, and parent education. Similar to in-home services used to
preserve families, reunification and stabilization services share the same
characteristics as other family-centered services, but are delivered in a more
intensive and targeted manner.
Auditors’ review of four cases where the families received reunification services
found that all four families were reunited with their children. However, for one of
these cases, the reunification lasted for only a short time. Specifically, while the
family received reunification services, the mother also received services through
the behavioral health and county probation systems for anger management and
to prevent a substance abuse relapse. Yet 2 weeks after the mother canceled
reunification services, the mother was arrested and jailed for violating her
intensive probation. According to case documentation, the mother’s probation
officer told CPS staff that he/she would not recommend reinstatement of regular
probation and that the mother could receive 1 to 1.5 years of jail time. Although
the mother was jailed, the child was still able to remain home with an adult sister.
In contrast, a 2009 evaluation of an April 2006 through December 2008 division
demonstration project to expedite reunifications through the use of contracted
intensive in-home and aftercare services found that the services were not
significantly more effective than standard CPS services at reunifying children
with their families, reducing children’s length of stay in out-of-home care,
decreasing the likelihood of re-entry into out-of-home care, and preventing the
recurrence of child abuse and neglect.2 CPS staff and in-home services
contractors described the expedited services as more intense, varied, and
1 Chaffin et al., 2001
2 Arizona State University, Center for Applied Behavioral Health Policy, 2009
page 12
State of Arizona
timely than standard CPS services. Literature on the effectiveness of similar
types of services has also indicated mixed results.1
Division redesigned in-home services contracts to help
strengthen the program
The Division redesigned the in-home services contracts to help strengthen the
in-home services program.2 According to division management, the changes made
to the contracts are based on best practice and include the following:
• More clearly defined service time frames and expectations for frequency
and type of contacts—In its redesigned contracts, the Division has clarified
time frames and expectations for service duration and the completion of
assessments and service plans to provide contractors with clearer guidance
and to promote accountability. In addition, the Division expanded the minimum
frequency of in-home visits and aligned them to the service duration time
frames. Home visits are a critical component of in-home services because they
provide an opportunity for contract staff to spend time with families and observe
them in their homes. During these visits, contract staff build relationships with
families that enable them to help the families more effectively respond to crises,
opportunities, and child and family needs.
• Peer mentors—The redesigned contracts also allow contractors to use parents
who have successfully completed CPS services and achieved reunification as
peer mentors. Peer mentors do not provide therapeutic treatment to parents, but
their similar backgrounds and their experiences successfully navigating the
child welfare system may offer hope that reunification and recovery are
achievable goals.3 Although research on the effectiveness of parent peer
mentors is limited, a 2009 study found that parents participating in a program
that paired them with parents who had successfully navigated the system were
more than four times as likely to be reunified with their children as parents in a
comparison group.4
1 Child Welfare Information Gateway, 2011b; Fraser, Walton, Lewis, Pecora, & Walton, 1996; Lewandowski & Pierce,
2002; Littell & Schuerman, 1995; and Walton, 1998
2 Although the Department of Economic Security (Department) released the in-home services contract solicitation in
December 2010, in accordance with Arizona Administrative Code R2-7-A902, it postponed implementation of the
contracts soon after awarding them in May 2011 so that the Department could review and render a decision on several
bidder protests regarding the procurement process. Four entities appealed the Department’s decisions to the
Department of Administration (DOA). As of March 1, 2012, one entity’s protest was still pending a decision by DOA.
Further, another entity requested a rehearing of DOA’s decision. To ensure that in-home services were not disrupted as
a result of the postponed contracts, the Department extended the existing in-home services contracts that would have
terminated in May 2011 pending final resolution of the protests.
3 Frame, Berrick, & Knittel, 2010
4 Anthony, Berrick, Cohen, & Wilder, 2009
page 13
Office of the Auditor General
• Evidence-based practices—Further, the redesigned contracts
require contractors to have evidence-based practices available in
several service areas, including parent education and training, crisis
intervention, and counseling (see textbox for definition of evidence-based
practices). The use of evidence-based practices provides
greater assurance that children and families are receiving effective
services to address their needs. Literature indicates that providing
services that are evidence-based may yield better results.1
Additionally, the federal government and states are emphasizing the
use of evidence-based practices in response to a need for greater
effectiveness and accountability of programs serving children and
families.2 See textbox for examples of evidence-based practices.
Improving outcomes for families, which may be achieved through the use of
evidence-based practices, can also impact the ability of organizations to obtain
and sustain program funding. Not only do legislatures, foundations, and other
funding entities increasingly want to invest their dollars in programs that have
demonstrated their effectiveness through research, but the public also wants to
know that tax dollars are being spent on programs and services that actually
work.3 For example, the federal government is interested in funding programs
that have shown “...sizeable, sustained effects on important child outcomes
1 Cooney, Huser, Small, & O’Connor, 2007; Kumpfer & Alvarado, 2003; and Small, Cooney, & O’Connor, 2009
2 This demand for greater effectiveness and accountability not only affects the field of child welfare, but also the juvenile
justice and mental health fields as they use many similar practices, including case management, counseling/therapy,
skill building, and provision of concrete services such as food and transportation.
3 Small et al., 2009
Examples of evidence-based practices
Trauma-focused cognitive behavioral therapy—A treatment intervention shown to reduce child acting-out
behavior resulting from exposure to traumatic life events such as child sexual abuse and exposure to domestic
violence. Therapy sessions can be conducted in various settings, including the family’s home.
Parent-child interaction therapy—A treatment program shown to reduce the recurrence of physical abuse of
young children with conduct disorders. Therapy sessions are typically conducted in a community agency or
outpatient clinic.
Brief Strategic Family Therapy®—A therapy shown to improve family functioning, including effective parental
leadership and management, positive parenting, and parental involvement with the child and his/her peers and
school. Therapy sessions are conducted at locations convenient to the family, including the family’s home.
Homebuilders®—A program shown to reduce the recurrence of out-of-home placements. Intensive in-home crisis
intervention, counseling, and life-skills education are provided to families who have children at imminent risk of
placement in state-funded care.
Source: CEBC, 2011; SAMHSA's NREPP, 2008; CEBC, 2009; SAMHSA's NREPP, 2009; University of Miami, Miller School of Medicine, n.d.; Institute
for Family Development, 2010
Evidence-based practice
means using an intervention,
program, or treatment that
has been established as
effective through scientific
research according to a set
of explicit criteria.
Source: Lederman, Gómez-Kaifer,
Katz, Thomlison, & Maze,
2009
page 14
State of Arizona
such as abuse and neglect” and is passing
laws to encourage the use of evidence-based
practices (see textbox).1
Although there may be initial costs
associated with the adoption and
implementation of evidence-based
practices and some evidence-based
practices may have higher treatment costs
for children and families than unproven
services, some have been shown to be
cost effective. For example, as part of a
2008 study by the Washington State Institute
for Public Policy (Institute) to determine
whether evidence-based programs and
policies could reduce the likelihood of
children entering and remaining in the child
welfare system, the Institute examined
whether the benefits of evidence-based programs outweighed program costs.
The Institute found several evidence-based programs could generate long-term
monetary benefits in excess of program costs. For example, Homebuilders®,
an intensive family preservation program, had a net benefit of $2.54, and parent-child
interaction therapy had a net benefit of $5.93 for every dollar spent.2,3 The
net benefits were estimates of the economic benefits expected to accrue on
outcomes that could be monetized, specifically, child abuse and neglect, out-of-
home placement, crime, education, substance abuse, teen pregnancy, and
public assistance.
Division should further support contractors’
implementation of evidence-based practices
Although the Division’s in-home services contracts that were awarded in May 2011
require that contractors have evidence-based, in-home services available, the
Division should take additional steps to support their use. First, the Division should
communicate its intent that in-home services be evidence-based and establish this
as a requirement in its next contract solicitation. The Division should also develop
well-defined, written criteria that can be used to identify evidence-based practices
and maintain an updated inventory of these practices. Finally, the Division should
1 PL 110-161, Division G, Title II, p. 1540-1541.
2 Lee, Aos, & Miller, 2008
3 In July 2011, the Washington State Institute for Public Policy issued updated cost-benefit information showing a net
benefit of $3.41 for the Homebuilders® program and $6.27 for parent child interaction therapy (see bibliography).
Patient Protection and Affordable Care Act
• Provided $1.5 billion in new funds over 5 years for early
childhood home visitation programs, with at least 75 percent of
the funds used for evidence-based programs.
• Provided $75 million annually for 5 years for programs that
replicate evidence-based teen pregnancy prevention strategies
and incorporate other adult responsibility subjects, such as
maintaining healthy relationships, improving communication with
parents, and financial literacy.
Consolidated Appropriations Act, 2010
• Provided $110 million in new funds for teenage pregnancy
prevention programs, with at least $75 million used for
evidence-based programs.
Source: P.L. 111-148, §§2951 & 2953; P.L. 111-117, Division D, Title II, p. 3252-3253.
page 15
Office of the Auditor General
expand its monitoring of in-home services contractors to ensure they are implementing
evidence-based practices as designed.
Division should communicate its intent that contracted in-home
services be evidence based and require this in next contract
solicitation—To support the use of evidence-based practices in the in-home
services program, the Division should communicate its intent to its stakeholders,
including existing and potential contractors, that in-home services be based on
evidence-based practices and then make this a requirement in the next contract
solicitation. The in-home services contracts awarded in May 2011, but not yet
implemented as of March 1, 2102, require that contractors have evidence-based
practices available in several service areas, including parent education and
training, but their use is optional. According to division management, the use of
evidence-based practices was not required because of concerns regarding their
potential cost. For example, Homebuilders® is an evidence-based program that
has been shown to effectively reduce out-of-home placements (see textbox on
page 13). However, the caseload is one to two families per therapist, making it a
resource-intensive program.
Because these contracts have an initial contract term of 1 year with four 1-year
renewal options, this allows existing and potential contractors time to develop or
expand their capacity to implement evidence-based, in-home services. According
to the National Implementation Research Network, implementation is a process
that takes 2 to 4 years to complete in most provider organizations and involves
several stages during which critical functions of implementation must be
addressed, including practitioner training, coaching the practitioner on the job,
regularly assessing whether the program is implemented as designed, and using
that information to improve the performance of practitioners who are carefully
selected for their positions.1
The time between contract solicitations also provides the Division with time to
develop its in-house expertise on evidence-based practices relevant to in-home
services and explore options to support existing and future contractors in their
implementation of these practices. For example, designated division staff should
become familiar with available evidence-based practices that are relevant to the
types of issues and children and families generally served through the program.
In addition, the Division should explore ways to broadly disseminate this
information to its stakeholders. The Division should also explore potential public
and private funding that might be available for implementing evidence-based
practices and share this information with its stakeholders, or, if appropriate, apply
for it directly. For example, in June 2011, the Children’s Bureau posted a funding
announcement for integrating trauma-informed and trauma-focused practice into
1 Fixsen, Naoom, Blase, Friedman, & Wallace, 2005
page 16
State of Arizona
child protective services delivery.1 The purpose of the grant was to provide an
opportunity for child welfare systems to introduce one or more evidence-based or
evidence-informed, trauma-focused treatments into their service arrays (see
textbox on page 13 for a description of an evidence-based, trauma-focused
practice). The estimated grant was $3.2 million, and it was expected that five
5-year awards would be made. Eligible applicants are state, tribal, or county public
child welfare agencies and private child welfare agencies under contract with the
public child welfare agency.
Therefore, the Division should require the use of evidence-based practices in the
next contract solicitation for in-home services. In the meantime, the Division should
communicate this intent to allow existing and potential contractors time to develop
or expand their capacity to implement evidence-based in-home services. The
Division should also begin its preparations to make the use of evidence-based
practices a requirement in the next contract solicitation by developing its in-house
expertise on evidence-based practices relevant to in-home services. For example,
designated division staff should become familiar with available evidence-based
practices that are relevant to the types of issues and children and families gener-ally
served through the program.
Division should establish well-defined criteria to identify evidence-based
practices and maintain an updated inventory of these
practices—To help prepare for the use of evidence-base practices in its
in-home services program, the Division should develop well-defined, written
criteria for identifying appropriate evidence-based practices and maintain an
updated inventory of these practices. This information should be made available
through the Department’s Web site. The Division’s awarded but not yet
implemented in-home services contracts define evidence-based practices, but do
not include clear criteria for identifying these practices. Specifically, the contracts
define evidence-based practice as “practice which incorporates careful
consideration of current research, and the provision of relevant, non-biased, and
comprehensive information to provide best practice interventions with families.”
Although this definition provides general information for identifying evidence-based
practices, it may still result in a wide variation in the quality and effectiveness
of the services provided. For example, the Division’s definition requires “...careful
consideration of current research...” However, it does not address research quality,
which can vary significantly. A rigorous research design typically involves randomly
assigning participants to either the treatment group that participates in the
program or the control group that does not. This type of research design helps
ensure that any observed differences in outcomes between the two groups are the
result of the program and not the result of other factors. In contrast, less rigorous
1 The U.S. Department of Health and Human Service’s Children's Bureau has primary responsibility for administering
federal child welfare programs. The Children’s Bureau works in concert with states and tribes to provide for the safety,
permanency, and well-being of children. Through policy guidance, funding support, training, technical assistance, and
monitoring activities, the Children’s Bureau seeks to develop and disseminate knowledge, support comprehensive
systems change, and improve children’s lives.
page 17
Office of the Auditor General
research designs do not include any type of comparison group and, thus, do not
allow for any conclusions to be made about whether the changes seen in program
participants are related to or caused by the program.
Several government agencies, research organizations, and other associations
and national efforts have developed rating criteria for identifying evidence-based
practices. Typical criteria include a theoretical foundation for the practice, rigorous
evaluation, publication in a peer-reviewed journal, replication in different settings,
and implementation with fidelity to the original model. Programs and practices
meeting these criteria are compiled into registries of evidence-based practices,
such as the California Evidence-Based Clearinghouse for Child Welfare and the
Office of Juvenile Justice and Delinquency Prevention’s Model Programs Guide
(see Appendix A, pages a-i and a-ii, for examples of registries and databases).
Although these registries provide varying levels of information regarding evidence-based
practices, some registries provide a wealth of information, including a
general description of the practice, target population, expected outcomes,
evaluation studies reviewed to determine status as an evidence-based practice,
quality of research in the evaluation studies, readiness for dissemination, cost, and
contact information.
Therefore, the Division should develop well-defined, written criteria for identifying
evidence-based practices and maintain an updated inventory of these practices.
The Division should work with its in-home services contractors and other
knowledgeable sources, such as the National Resource Center for In-Home
Services, a national center of expertise regarding child welfare practice, to develop
the criteria. In addition, the Division should maintain an updated inventory of those
practices. The criteria and inventory of evidence-based practices should be made
available through the Department’s Web site to help existing and potential
contractors expand their capacity to provide evidence-based in-home services.
Division should expand its monitoring of in-home services contractors
to ensure evidence-based practices are implemented as
designed—As its contractors begin to incorporate evidence-based practices
into the services provided to division clients, the Division should expand its
monitoring of its contractors to ensure they implement the
evidence-based practices as designed. Although the Division
monitors in-home services contractors’ compliance with
contractual requirements, the current level of monitoring does
not gather sufficient information to ensure evidence-based
practices are implemented as designed. For example, the
Division does not monitor whether therapies/curricula are
being delivered appropriately and to the populations for which
they were intended.
“Desirable outcomes are achieved
only when effective programs are
implemented well.”
Source: Fixsen et al., 2005
page 18
State of Arizona
Ensuring that an intervention is provided as designed involves obtaining information
on how closely the implementation adheres to the practice’s essential components,
including staffing, training, content, and program delivery.1 This can be done in a
number of ways, including through checklists, client surveys, direct observation, and/
or videotaped observations. One of the most common monitoring methods is the
use of a checklist, log, or survey by the contractors, which places minimal burden on
contractors and is less costly than direct observation.2 A checklist can be used to
track specific aspects of implementing the evidence-based practice, including the
content covered, activities conducted, time spent conducting the activities, methods
for delivering the intervention, participant attendance, and participant responsiveness.
Some developers of evidence-based practices have created and made available
such checklists for their particular program. For example, a checklist exists for
trauma-focused cognitive behavioral therapy, which is a therapy that could potentially
be provided through the in-home services program.
Although implementing an evidence-based practice as designed is
important for achieving expected outcomes, some modifications to
the practice may be needed so that they will better fit the needs of
the family. However, substantial deviations from proven practices
can become problematic and should be avoided. At some point,
adaptation can render a practice so fundamentally different from
what the designers intended and what was studied that it can no
longer be considered evidence-based. There are typically two
components to an evidence-based practice—the essential
components and the adaptive components (see textbox).
Modifications should be limited to a practice’s adaptive components.
Therefore, the Division should expand its monitoring of in-home services contractors
to ensure they are implementing evidence-based practices as designed. Specifically,
the Division should ensure that contractors have implemented procedures to monitor
fidelity, whether through checklists, observations, client surveys, or some other
means. In addition, the Division should ensure that contractors have procedures to
correct deviations from an evidence-based practices’ design. If contractors modify
evidence-based practices, the Division should require them to provide written
justification for the modifications so that the Division can verify that essential
components are not being modified. The Division should also consider making
exceptions to modifications to essential components if the modification is approved
in writing by the practice’s developer(s).
1 Gorman-Smith, 2006
2 James Bell Associates, 2009
Essential components are those
program components that are linked
through theory or research to positive
outcomes and program effectiveness.
Adaptive components are program
features that are optional or can be
modified to fit the resources and
needs of the community without
impacting program effectiveness.
Source: James Bell Associates, 2009
page 19
Office of the Auditor General
Recommendations:
1.1 The Division should require the use of evidence-based practices in the next
contract solicitation for in-home services.
1.2 In the meantime, the Division should communicate its intent to its stakeholders,
including existing and potential contractors, for requiring the use of evidence-based
practices to allow time for existing and potential contractors to develop
or expand their capacity to provide evidence-based in-home services.
1.3 The Division should use the time until the next contract solicitation for in-home
services to:
a. Develop its in-house expertise in order that it may effectively support
contractors’ implementation of evidence-based in-home services,
b. Develop and make available through the Department’s Web site well-defined,
written criteria for identifying evidence-based practices, and
c. Maintain and make available through the Department’s Web site an
updated inventory of evidence-based practices.
1.4 The Division should expand its monitoring of in-home services contractors to:
a. Ensure the contractors are implementing evidence-based practices as
designed, and
b. Ensure that contractors have procedures to correct deviations from
evidence-based practices’ design.
1.5 The Division should require contractors modifying evidence-based practices to
provide written justification for the modifications to verify that essential
components are not being modified without approval of the developer(s).
page 20
State of Arizona
Office of the Auditor General
Evidence-based practice: Selected
registries and databases
Several government
agencies, research
organizations, and other
associations and national
efforts have developed
rating criteria for identifying
evidence-based practices.
Typical criteria include a
theoretical foundation for
the practice, rigorous
evaluation, publication in a
peer-reviewed journal,
replication in different
settings, and
implementation with fidelity
to the original model.
Programs and practices
meeting these criteria are
compiled into registries and
databases of evidence-based
practices. Although
these registries provide
varying levels of information
regarding the evidence-based
practices, some
registries provide a wealth
of information, including a
general description of the
practice, target population,
expected outcomes,
evaluation studies reviewed
to determine status as an
evidence-based practice,
quality of research in the
evaluation studies,
readiness for
dissemination, cost, and
contact information.
California Evidence-Based Clearinghouse for Child Welfare (CEBC)
http://www.cebc4cw.org/
The CEBC provides child welfare professionals with easy access to vital
information about selected child welfare-related programs. Each program
is reviewed and rated using the CEBC scientific rating scale to determine
the level of evidence for the program. The programs are also rated on a
relevance to child welfare rating scale.
The Campbell Collaboration Library of Systematic Reviews
http://www.campbellcollaboration.org/library.php
The Campbell Library of Systematic Reviews (Library) provides free online
access to systematic reviews, titles, protocols, and user abstracts in the
areas of education, criminal justice, and social welfare. The Library is a
peer-reviewed source of reliable evidence of the effects of interventions.
CDC: The Community Guide
http://www.thecommunityguide.org/
The Guide to Community Preventive Services (Community Guide)
summarizes what is known about interventions’ effectiveness, economic
efficiency, and feasibility to promote community health and prevent disease.
The Task Force on Community Preventive Services makes recommendations
for the use of various interventions based on the evidence gathered in the
rigorous and systematic scientific reviews of published studies conducted
by the review teams of the Community Guide. The findings from the reviews
are published in peer-reviewed journals and also made available on this
Web site.
National Registry of Evidence-based Programs and Practices (NREPP)
http://www.nrepp.samhsa.gov/
The NREPP is a searchable database of interventions for the prevention
and treatment of mental and substance use disorders. The Substance
Abuse and Mental Health Services Administration (SAMHSA) has developed
this resource to help people, agencies, and organizations implement
programs and practices in their communities.
page a-i
APPENDIX A
The Office of Juvenile Justice and Delinquency Prevention’s Model Programs Guide (MPG)
http://www.ojjdp.gov/mpg/
The MPG is designed to assist practitioners and communities implement evidence-based prevention and
intervention programs that can make a difference in the lives of children and communities. The MPG is an
easy-to-use tool that offers a database of scientifically proven programs that address a range of issues,
including substance abuse, mental health, and education programs.
SAMHSA: A Guide to Evidence-Based Practices on the Web
http://www.samhsa.gov/ebpwebguide/
SAMHSA provides this Web guide to assist the public with simple and direct connections to Web sites that
contain information about interventions to prevent and/or treat mental and substance use disorders. The
Web guide provides a list of Web sites that contain information about specific evidence-based practices or
provide comprehensive reviews of research findings.
Social Programs that Work
http://www.evidencebasedprograms.org/
This Web site summarizes the findings from well-designed, randomized, controlled trials that, in their view,
have particularly important policy implications because they show, for example, that a social intervention has
a major effect or that a widely used intervention has little or no effect. They limit the discussion to well-designed,
randomized, controlled trials based on persuasive evidence that they are superior to other study
designs in measuring an intervention’s true effect.
State of Arizona
page a-ii
Office of the Auditor General
Bibliography
This bibliography includes,
among others, citations on
literature reviewed by
auditors on the
effectiveness of services to
help preserve, support, and
reunify/stabilize families at
risk for child abuse and
neglect.
American Public Human Services Association. (2005). Guide for child welfare
administrators on evidence based practice. Washington, DC: Author.
Anthony, E. K., Berrick, J. D, Cohen, E., & Wilder, E. (2009). Partnering with
parents: Promising approaches to improve reunification outcomes for
children in foster care. Berkeley, CA: University of California, Center for
Social Services Research.
Aos, S., Lee, S., Drake, E., Pennucci, A., Klima, T., Miller, M., et al. (2011).
Return on investment: Evidence-based options to improve statewide
outcomes: Technical appendix I detailed tables. Olympia, WA:
Washington State Institute for Public Policy.
Arizona State University, Center for Applied Behavioral Health Policy. (2009).
Arizona IV-E waiver expedited reunification demonstration: Final report April
1, 2006 – December 31, 2008. Phoenix, AZ: Author.
Barth, R. (2009). Preventing child abuse and neglect with parent training:
Evidence and opportunities. The Future of Children, 19(2), 95-118.
Barth, R. (2008). The move to evidence-based practice: How well does it fit
child welfare services? Journal of Public Welfare, 2(2), 145-171.
Berry, M., Propp, J., & Martens, P. (2007). The use of intensive family
preservation services with adoptive families. Child and Family Social
Work, 12, 43-53.
Biehal, N. (2005). Working with adolescents at risk of out of home care: The
effectiveness of specialist teams. Children and Youth Services Review, 27,
1045-1059.
Blythe, B., & Jayaratne, S. (2002). Michigan Families First effectiveness study.
Lansing, MI: State of Michigan, Department of Human Services.
California Evidenced-Based Clearinghouse for Child Welfare. (2011). Trauma-focused
cognitive behavioral therapy. Retrieved August 19, 2011, from
http://www.cebc4cw.org/program/trauma-focused-cognitive-behavioral-therapy/
detailed
California Evidence-Based Clearinghouse for Child Welfare. (2009). Parent-child
interaction therapy. Retrieved August 24, 2011, from http://www.cebc4cw.
org/program/parent-child-interaction-therapy/detailed
Cash, S., & Berry, M. (2003). The impact of family preservation services on child
and family well-being. Journal of Social Service Research, 29(3), 1-26.
Chaffin, M., Bonner, B., & Hill. R. (2001). Family preservation and family support
programs: Child maltreatment outcomes across client risk levels and
program types. Child Abuse & Neglect, 25, 1269-1289.
Chaffin, M. & Friedrich, B. (2004). Evidence-based treatments in child abuse
and neglect. Children and Youth Services Review, 26, 1097-1113.
APPENDIX B
page b-i
page b-ii
State of Arizona
Child Welfare Information Gateway. (2011a). Child maltreatment prevention: Past, present,
and future. Washington, DC: U.S. Department of Health and Human Services,
Administration for Children and Families, Children’s Bureau.
Child Welfare Information Gateway. (2011b). Family reunification: What the evidence
shows. Washington, DC: U.S. Department of Health and Human Services,
Administration for Children and Families, Children’s Bureau.
Cooney, S. M., Huser, M., Small, S. A., & O’Connor, C. (2007). Evidence-based
programs: An overview. (What works, Wisconsin Research to Practice Series, Issue
6). Madison, WI: University of Wisconsin-Madison and University of Wisconsin—
Extension.
Dagenais, C., Begin, J., Bouchard, C., & Fortin, D. (2004). Impact of intensive family
support programs: A synthesis of evaluation studies. Children and Youth Services
Review, 26, 249-263.
Duggan, A., Caldera, D., Rodriguez, K., Burrell, L., Rhode, C., & Crowne, S.S. (2007).
Impact of a statewide home visiting program to prevent child abuse. Child Abuse
& Neglect, 31, 801-827.
Durlak, J.A. & DuPre, E.P. (2008). Implementation matters: A review of research on the
influence of implementation on program outcomes and the factors affecting
Implementation. American Journal of Community Psychology, 41, 327-350.
Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005).
Implementation research: A synthesis of the literature [FMHI Publication #231].
University of Southern Florida, Louis de la Parte Florida Mental Health Institute,
National Implementation Research Network.
Frame, L., Berrick, J.D., & Knittel, J. (2010). Parent mentors in child welfare: A paradigm
shift from traditional services. The Source, 20(1), p. 2-6.
Fraser, M.W., Walton, E., Lewis, R.E., Pecora, P.J., & Walton, W.K. (1996). An experiment
in family reunification: Correlates of outcomes at one-year follow-up. Children and
Youth Services Review, 18(4/5), 335-361.
Freundlich, M. (2010). Legislative strategies to safely reduce the number of children in
foster care. Denver, CO: National Conference of State Legislatures.
Gorman-Smith, D. (2006). How to successfully implement evidence-based social
programs: A brief overview for policymakers and program providers [working
paper]. Washington, DC: Coalition for Evidence-Based Policy.
Heneghan, A. M., Horwitz, S.M., & Leventhal, J.M. (1996). Evaluating intensive family
preservation programs: A methodological review. Pediatrics, 97(4), 535-542.
Howard, K.S., & Brooks-Gunn, J. (2009). The role of home-visiting programs in
preventing child abuse and neglect. The Future of Children, 19(2), 119-146.
Howing, P.T., Wodarski, J.S., Gaudin, Jr., J.M., & Kurtz, P.D. (1989). Effective interventions
to ameliorate the incidence of child maltreatment: The empirical base. Social Work,
330-338.
page b-iii
Office of the Auditor General
Institute for Family Development. (2010). Intensive family preservation service and
intensive family reunification services. Retrieved September 12, 2011, from http://
www.institutefamily.org/programs_IFPS.asp
James Bell Associates (2010). Implementation resource guide for social service
programs: An introduction to evidence-based programs. Washington, DC: U S
Department of Health & Human Services, Administration for Children and Families,
Office of Family Assistance.
James Bell Associates (2009). Evaluation brief: Measuring implementation fidelity.
Arlington, VA: Author.
Kahn, J., Moore, B.A., & Moore, K. A. (2010). What works for home visiting programs:
Lessons from experimental evaluations of programs and interventions [Child Trends
Fact Sheet Publication #2010 20008]. Washington, DC: Child Trends.
Kauffman Best Practices Project. (2004). Closing the quality chasm in child abuse
treatment: Identifying and disseminating best practices: Findings of the Kauffman
best practices project to help children heal from child abuse. San Diego, CA:
Children’s Hospital-San Diego.
Kimberlin, S.E., Anthony, E.K., & Austin, M.J. (2009). Re-entering foster care: Trends,
evidence, and implications. Children and Youth Services Review, 31, 471–481.
Kirk, R.S., & Griffith, D.P. (2006). Annual report to the General Assembly of the State of
North Carolina on the Intensive Family Preservation Services Program for the 2005-
2006 state fiscal year. Raleigh, NC: General Assembly of the State of North
Carolina.
Kirk, R.S., & Griffith, D.P. (2004). Intensive family preservation services: Demonstrating
placement prevention using event history analysis. Social Work Research, 28(1),
5-16.
Klevens, J., & Whitaker, D. (2007). Primary prevention of child abuse and neglect. Child
Maltreatment, 12(4), 364-377.
Kumpfer, K. L., & Alvarado, R. (2003). Family-strengthening approaches for the
prevention of youth problem behaviors. American Psychologist, 58(6), 457–465.
Layzer, J.I., Goodson,B.D., Bernstein, L., & Price, C. (2001). National evaluation of family
support programs: Final report, Volume A: The meta-analysis. Washington, D.C.:
Administration for Children, Youth, and Families.
LeCroy, C.W., & Krysik, J. (In Press). Randomized trial of the healthy families Arizona
home visiting program. Children and Youth Services Review.
LeCroy & Milligan Associates, Inc. (2008). Arizona Promoting Safe and Stable Families
annual evaluation report: FFY 2007. Tucson, AZ: LeCroy & Milligan Associates, Inc.
Lederman, C., Gómez-Kaifer, M., Katz, L. E., Thomlison, B., & Maze, C. L. (2009). An
imperative: Evidence-based practice within the child welfare system of care.
Juvenile and Family Justice Today, 22-25.
Lee, S., Aos, S., & Miller, M. (2008). Evidence-based programs to prevent children from
entering and remaining in the child welfare system: Benefits and costs for
page b-iv
State of Arizona
Washington [Document No. 08-07-3901]. Olympia, WA: Washington State Institute
for Public Policy.
Lewandowski, C.A., & Pierce, L. (2002). Assessing the effect of family-centered, out-of-home
care on reunification outcomes. Research on Social Work Practice, 12, 205-
221.
Lietz, C. (2009). Examining families' perceptions of intensive in-home services: A mixed
methods study. Children and Youth Services Review, 31, 1337–1345.
Lindsey, D. (n.d.). Preserving families and protecting children: Finding the balance.
Retrieved March 29, 2010, from http://www.childwelfare.com/kids/fampres.htm
Lindsey, F., Martin, S., & Doh, J. (2002). The failure of intensive casework services to
reduce foster care placements: An examination of family preservation studies.
Children and Youth Services Review, 24(9-10), 743-775.
Littell, J.H. (1995). Evidence or assertions? The outcomes of family preservation services.
The Social Service Review, 69(2), 338-351.
Littell, J. H., & Schuerman, J.R. (2002). What works best for whom? A closer look at
intensive family preservation services. Children and Youth Services Review,
24(9/10), 673-699.
Littell, J. H., & Schuerman, J.R. (1995). A synthesis of research on family preservation and
family reunification programs. Washington, DC: United States Department of Health
and Human Services, Office of the Assistant Secretary for Planning and Evaluation.
Lundahl, B.W., Nimer, J., & Parson, B. (2006). Preventing child abuse: A meta-analysis of
parent training programs. Research on Social Work Practice, 16(3), 251-262.
MacLeod, J., & Nelson, G. (2000). Programs for the promotion of family wellness and the
prevention of child maltreatment: A meta-analytic review. Child Abuse & Neglect,
24(9), 1127-1149.
MacMillan, H. L., Wathen, C. N., Barlow, J., Ferguson, D. M., Leventhal, J. M., and
Taussig, H. N. (2009). Interventions to prevent child maltreatment and associated
impairment. The Lancet, 373, 250-266.
MacMillan, H.L. (2000). Preventive health care, 2000 update: Prevention of child
maltreatment. Canadian Medical Association Journal, 163(11), 1451-1458.
McCroskey, J., & Meezan, W. (1998). Family-centered services: Approaches and
effectiveness. The Future of Children, 8(1), 54-71.
McMurty, S. L. (1985). Secondary prevention of child maltreatment: A review. Social
Work, 42-48.
Miller, M. (2006). Intensive family preservation programs: Program fidelity influences
effectiveness—revised [Document No. 06-02-3901]. Olympia, WA: Washington
State Institute for Public Policy.
Nelson, K., Walters, B., Schweitzer, D., Blyth, B., & Pecora, P.J. (2009). A ten-year review
of family preservation research: Building the evidence base. Seattle, WA: Casey
Family Programs.
page b-v
Office of the Auditor General
O’Reilly, R., Wilkes, L., Luck, L., & Jackson, D. (2010). The efficacy of family support and
family preservation services on reducing child abuse and neglect: What the
literature reveals. Journal of Child Health Care, 14(1), 82-94.
Oshana, D. (2006). Evidence-based practice literature review. Chicago, IL: Prevent Child
Abuse America.
Pope, S. M., Williams, J. R., Sirles, E. A., & Lally, E. M. (2005). Family preservation and
support services: A literature review and report on outcome measures. Anchorage,
AK: The University of Alaska, School of Social Work, Anchorage Child Welfare
Evaluation program.
Research Review. (2007). Evidence-based programs and practices: What does it all
mean? Boynton Beach, FL: Children’s Services Council of Palm Beach County.
Reynolds, A.J., Mathieson, L.C., & Topitzes, J.W. (2009). Do early childhood interventions
prevent child maltreatment?: A review of the research. Child Maltreatment, 14(2),
182-206.
Roberts, A.R., & Everly, G.S. (2006). A meta-analysis of 36 crisis intervention studies.
Brief Treatment and Crisis Intervention, 6(1), 10-21.
Ryan, J.P., & Schuerman, J.R. (2004). Matching family problems with specific family
preservation services: A study of service effectiveness. Children and Youth Services
Review, 26,347-372.
Substance Abuse and Mental Health Services Administration, National Registry of
Evidence-Based Programs and Practices. (2009). Parent-child interaction therapy.
Retrieved August, 24, 2011, from http://nrepp.samhsa.gov/ViewIntervention.
aspx?id=23
Substance Abuse and Mental Health Services Administration, National Registry of
Evidence-based Programs and Practices. (2008). Brief strategic family therapy.
Retrieved August 24, 2011, from http://nrepp.samhsa.gov/ViewIntervention.
aspx?id=151
Substance Abuse and Mental Health Services Administration, National Registry of
Evidence-Based Programs and Practices. (2008). Trauma-focused cognitive
behavioral therapy. Retrieved August 24, 2011, from http://nrepp.samhsa.gov/
ViewIntervention.aspx?id=135
Small, S.A., Cooney, S.M., & O’Connor, C. (2009). Evidence-based program
improvement: Using principles of effectiveness to enhance the quality and impact
of family-based prevention programs. Family Relations, Interdisciplinary Journal of
Applied Family Studies, 58, 1-13.
Tully, L. (2008). Family preservation services literature review. Ashfield, NSW, Australia:
NSW Department of Community Service.
Tyuse, S.W., Hong, P.P., & Stretch, J.J. (2010). Evaluation of an intensive in-home family
treatment program to prevent out-of-home placement. Journal of Evidence-Based
Social Work, 7(3), 200-218.
page b-vi
State of Arizona
University of Miami, Miller School of Medicine. (n.d.). What is the BSFT™ Program?
Retrieved August 24, 2011, from http://bsft.org/art/what_is_bsft
U.S. Department of Health and Human Services, Administration for Children and
Families, Administration on Children, Youth and Families, Children’s Bureau.
(2011). Child maltreatment 2010. Washington, DC: Author.
U.S. Department of Health and Human Services, Administration for Children and
Families, Administration on Children, Youth and Families, Children’s Bureau.
(2010). Child maltreatment 2009. Washington, DC: Author.
U.S. Department of Health and Human Services, Administration for Children and
Families, Administration on Children, Youth and Families, Children’s Bureau.
(2009).Child maltreatment 2008. Washington, DC: Author.
U.S. Department of Health and Human Services, Administration for Children and
Families, Administration on Children, Youth and Families, Children’s Bureau.
(2008). Child maltreatment 2007. Washington, DC: Author.
U.S. Department of Health and Human Services, Administration for Children and
Families, Administration on Children, Youth and Families, Children’s Bureau.
(2007). Child maltreatment 2006. Washington, DC: Author.
U.S. Department of Health and Human Services, Administration for Children and
Families, Administration on Children, Youth and Families, Children’s Bureau.
(2006). Child maltreatment 2005. Washington, DC: Author.
Waldfogel, J. (2009). Prevention and the child protection system. Preventing Child
Maltreatment, 19(2), 195-210.
Wells, K., & Whittington, D. (1993). Child and family functioning after intensive family
preservation services. Social Service Review, 55-83.
Westat, Inc., James Bell Associates, Inc., & Chapin Hall Center for Children at the
University of Chicago. (2002). Evaluation of family preservation and reunification
programs: Final report. Washington, DC: U.S. Department of Health & Human
Services, Assistant Secretary for Planning and Evaluation.
Westat, Inc., James Bell Associates, Inc., & Chapin Hall Center for Children at the
University of Chicago. (1995). A review of family preservation and family reunification
programs. Washington, DC: U.S. Department of Health & Human Services, Office
of the Assistant Secretary for Planning and Evaluation.
AGENCY RESPONSE
AGENCY RESPONSE
CPS Reports Issued
Information Briefs
IB-0401 DES’ Federal IV-E Waiver
Demonstration Project
Proposal
IB-0501 Family Foster Homes and
Placements
IB-0502 Revenue Maximization
IB-601 In-Home Services Program
IB-0701 Federal Deficit
Reduction Act of 2005
IB-0702 Federal Grant Monies
IB-0801 Child Removal Process
IB-0901 CPS Client Characteristics
Questions and Answers
QA-0601 Substance-Exposed
Newborns
QA-0701 Child Abuse Hotline
QA-0702 Confidentiality of CPS
Information
QA-0703 Licensed Family Foster
Homes
QA-0801 Child and Family
Advocacy Centers
QA-0802 Processes for Evaluating and
Addressing CPS Employee
Performance and Behavior
QA-0901 Adoption Program
QA-1001 CPS Central Registry
Performance Audits
CPS-0501 CHILDS Data Integrity
Process
CPS-0502 Timeliness and
Thoroughness of
Investigations
CPS-0601 On-the-Job Training and
Continuing Education
CPS-0701 Prevention Programs
CPS-0701 Prevention Programs
CPS-0801 Complaint Management
Process
CPS-0901 Congregate Care
CPS-0902 Relative Placement
CPS-1101 Contractor Payments