Performance Audit Family Builders Program

              State of Arizona
Office
of the
Auditor General
PERFORMANCE AUDIT
Report to the Arizona Legislature
By Debra K. Davenport
Auditor General
FAMILY BUILDERS
PROGRAM
November 2001
Report No. 01-30
The Auditor General is appointed by the Joint Legislative Audit Committee, a bipartisan committee
composed of five senators and five representatives. Her mission is to provide independent and impar-tial
information and specific recommendations to improve the operations of state and local government
entities. To this end, she provides financial audits and accounting services to the state and political
subdivisions and performance audits of state agencies and the programs they administer.
The Joint Legislative Audit Committee
Senator Ken Bennett, Chairman
Representative Roberta L. Voss, Vice-Chairman
Senator Herb Guenther Representative Robert Blendu
Senator Dean Martin Representative Gabrielle Giffords
Senator Peter Rios Representative Barbara Leff
Senator Tom Smith Representative James Sedillo
Senator Randall Gnant (ex-officio) Representative James Weiers (ex-officio)
Audit Staff
Carol Cullen—Manager
and Contact Person (602) 553-0333
Tanya Nieri—Team Leader
Terri Place—Team Member
Copies of the Auditor General’s reports are free.
You may request them by contacting us at:
Office of the Auditor General
2910 N. 44th Street, Suite 410
Phoenix, AZ 85018
(602) 553-0333
Additionally, many of our reports can be found in electronic format at:
www.auditorgen.state.az.us
2910 NORTH 44th STREET • SUITE 410 • PHOENIX, ARIZONA 85018 • (602) 553-0333 • FAX (602) 553-0051
DEBRA K. DAVENPORT, CPA
AUDITOR GENERAL
STATE OF ARIZONA
OFFICE OF THE
AUDITOR GENERAL
WILLIAM THOMSON
DEPUTY AUDITOR GENERAL
November 1, 2001
Members of the Arizona Legislature
The Honorable Jane Dee Hull, Governor
Mr. John Clayton, Director
Department of Economic Security
Transmitted herewith is a report of the Auditor General, An Evaluation of the Family
Builders Program administered by the Arizona Department of Economic Security. This
evaluation was conducted pursuant to Laws 2000, Ch. 23, §2. I am also transmitting
with this report a copy of the Report Highlights for this evaluation to provide a quick
summary for your convenience.
As outlined in its response, the Department of Economic Security plans to implement
all of the recommendations.
My staff and I will be pleased to discuss or clarify items in the report.
This report will be released to the public on November 2, 2001.
Sincerely,
Debra K. Davenport
Auditor General
Enclosure
i
OFFICE OF THE AUDITOR GENERAL
SUMMARY
The Office of the Auditor General has completed an evaluation
of the Family Builders Program. This evaluation was conducted
pursuant to the provisions of Laws 2000, Ch. 23, §2, and provides
information regarding the program’s effectiveness in achieving
its goals.
Family Builders is a family-centered, community-based program
that provides early intervention to families with potential- and
low-risk child abuse or neglect reports made to Child Protective
Services (CPS). The program’s goal is to enhance parents’ ability
to create safe, stable, and nurturing home environments that
promote the safety of all family members and healthy child de-velopment.
CPS refers potential- and low-risk reports of child
abuse and neglect to community-based Family Builders contrac-tors.
These contractors provide services, such as parenting skills
training, counseling, and utilities assistance, aimed at reducing
the problems contributing to the potential for abuse or neglect.
Participation in the program is voluntary, and families typically
participate for about 6 months.
The Arizona Department of Economic Security (Department) is
responsible for administering Family Builders. To do this, the
Department contracts with eight community-based organiza-tions
to provide services to families in ten Arizona counties.
Contractors are paid a capitated rate for each of the three
program phases completed by a family: referral, assessment, and
service plan.
Family Builders received 14,249 referrals between August 1, 1999
and April 30, 2001. In 9,586 of these cases, families completed
only the referral phase. In 4,663 cases, families received an as-sessment.
In all but 266 of the assessed cases, families signed a
service plan to receive services.
Summary
ii
OFFICE OF THE AUDITOR GENERAL
Services Differ but
Program Outcomes
Comparable to CPS
(See pages 13 through 18)
Although the services provided to families differ, Family Build-ers
and CPS had similar outcomes related to subsequent CPS re-ports.
Family Builders offers more services to low-risk families
than CPS does, yet evaluators found that the proportion of fami-lies
served by Family Builders who received subsequent CPS re-ports
was comparable to the proportion of families investigated
by CPS who received subsequent reports. Evaluators also ana-lyzed
the number of CPS reports per 10,000 persons for areas
served by Family Builders and areas served only by CPS and
found them to be similar.
Despite these similar outcomes, positive differences exist among
program participants. Families who completed the program had
fewer subsequent CPS reports than families who did not com-plete
the program. Further, families receiving program services
experienced a slight reduction in their risk for child abuse and
neglect, as measured by the caseworker-completed Family Risk
Scale.
Department Has Made
Limited Progress in
Monitoring and Oversight
(See pages 19 through 24)
Although the Department has taken steps to address monitoring
and oversight problems identified in the Auditor General’s
March 2000 evaluation (see Report No. 00-4), its progress has
been limited. The Department still needs to improve its cost data
for it to make statements about the type and cost of services de-livered
and ensure that capitation rates are appropriate. In a ran-dom
sample of 100 cases, evaluators found many problems, such
as missing and miscategorized costs. Although the Department
has recently revised its cost category definitions, which had been
a primary cause of problems, the database still contains data en-tered
according to old definitions. The Department also still
Summary
iii
OFFICE OF THE AUDITOR GENERAL
needs to improve other program data, such as demographic and
risk assessment information, and to eliminate problems includ-ing
missing and duplicative records. Although the Department
has taken some steps to address the Family Builders database
flaws and other factors contributing to the persistence of data
problems, additional effort is needed. Specifically, the Depart-ment
should develop a data quality assurance plan and conduct
a system-wide review of the Family Builders database to elimi-nate
current problems. It should also monitor contractor’s use of
the recently modified cost categories and program definitions,
create a database users’ manual, and provide regular training.
Finally, the Department should assign the technical resources
required to effectively address current problems and maintain
the data collection system.
Statutory Evaluation
Components
(See pages 25 through 35)
As required by Laws 2000, Ch. 23, §2, the Office of the Auditor
General has included in this report information on the number of
referrals to the program, the number of families served, partici-pants’
demographic characteristics, the services delivered, client
satisfaction, the cost of services, the contractors who provided
services, recommended improvements in program administra-tion,
the extent to which program goals and objectives are being
met, and the outcomes for families served by the program, in-cluding
its impact on reducing the risk of child maltreatment.
iv
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
v
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS
Page
Introduction and Background............................. 1
Finding I: Services Differ but
Program Outcomes
Comparable to CPS......................................... 13
Family Builders and CPS
Had Similar Outcomes....................................................... 13
Positive Differences Exist
Among Program Participants............................................ 17
Finding II: Department Has Made
Limited Progress in Monitoring
and Oversight.................................................. 19
Improvement Still Needed
on Cost Data....................................................................... 19
Improvement Also Still Needed
on Other Program Data ..................................................... 20
Contractors Continue To Need
Guidance in Operating Local
Advisory Boards ................................................................ 23
Recommendations ............................................................. 24
Statutory Evaluation Components ...................... 25
Appendix............................................................... a-i
Agency Response
Table of Contents
vi
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS (Cont’d)
Page
Figures
Figure 1 Family Builders Program
Contractors by County
As of June 2001................................................. 3
Figure 2 Family Builders Program Process
As of June 2001................................................. 5
Figure 3 Family Builders Program
Primary Caregivers’ Marital Status................. 26
Figure 4 Family Builders Program
Primary Caregivers’ Highest Level
of Education Completed.................................. 27
Tables
Table 1 Family Builders Program
Statement of Revenues and Expenditures
Years Ended or Ending June 30, 2000,
2001, and 2002 (Unaudited)............................. 8
Table 2 Family Builders Program
Contractors, Number of Referrals,
and Percentage of Cases
With Assessments and Service Plans
August 1, 1999 through April 30, 2001............ 9
Table of Contents
vii
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS (Concl’d)
Page
Tables (Concl’d)
Table 3 Family Builders Program
Families with Subsequent CPS Reports
Family Builders Participants vs.
Families Investigated by CPS
January 1, 1998 through March 31, 2001 ......... 16
Table 4 Family Builders Program
Family Risk Scale at Entrance
and Exit Scores of Families with
Closed Service Plan Cases
January 1, 1998 through
September 30, 2000........................................... 18
Table 5 Family Builders Program
Family Risk Scale Assessment
Areas and Scores.............................................. a-ii
viii
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
1
OFFICE OF THE AUDITOR GENERAL
INTRODUCTION AND BACKGROUND
The Office of the Auditor General has completed an evaluation
of the Family Builders Program conducted pursuant to the pro-visions
of Laws 2000, Ch. 23, §2. This is the second evaluation
since the program’s beginning in 1998. A third evaluation is due
in 2004. This report provides information about the program, ex-amines
various aspects of the program’s effectiveness, and rec-ommends
ways to improve its administration.
Family Builders Program
History and Background
The Legislature established the Family Builders Pilot Program in
the Department of Economic Security, Division of Children,
Youth and Families (hereafter referred to as the Department)
through Laws 1997, Ch. 223, §2. The pilot program was estab-lished
to address a backlog of uninvestigated child abuse and
neglect reports and to fill a need for early intervention services.
During fiscal year 1997, the Department’s Child Protective Ser-vices
(CPS) function did not have enough investigators to re-spond
adequately to an increased number of child abuse and ne-glect
reports. Though high- and moderate-risk reports were be-ing
investigated, a backlog of 5,899 reports, most of which were
low and potential risk, remained at the end of 1997. The Family
Builders Pilot Program was implemented in Maricopa and Pima
Counties to respond to these reports and to help prevent subse-quent
child abuse and neglect through early intervention. Since
the inception of Family Builders, the Department has been able
to respond to 100 percent of all reports, either by investigating
the cases or referring them to Family Builders for services.
The Office of the Auditor General’s March 2000 evaluation found
that although families participating in the program showed a
slight improvement in their risk for child abuse and neglect, they
were just as likely to have a new CPS report after entering the
program as were two comparison groups of families not partici-pating
in the program.
The previous evaluation
found that program fami-lies
were as likely as com-parison
group families to
have a subsequent CPS
report.
Introduction and Background
2
OFFICE OF THE AUDITOR GENERAL
In October 1999, the Department expanded the original service
area of the Family Builders Pilot Program, which comprised only
Maricopa and Pima Counties, to include Apache, Cochise, Co-conino,
Graham, Greenlee, Navajo, Santa Cruz, and Yavapai
Counties. Figure 1 (see page 3) shows the counties served by
Family Builders and the contractors that serve them. In the five
counties that are not served by Family Builders—Gila, La Paz,
Mohave, Pinal, and Yuma— CPS investigates all reports received
on its hotline. Through A.R.S. §8-816, effective June 30, 2000, the
Legislature extended the program for 10 years and renamed it
the Family Builders Program.
Family Builders: A Family-Centered,
Community-Based Alternative to CPS
Family Builders is a family-centered, community-based alterna-tive
to CPS investigations. It is based on the premise that families
with different risk levels for child abuse and neglect require dif-ferent
responses from CPS. While the serious nature of moder-ate-
to high-risk reports often requires a CPS investigation, low-and
potential-risk reports can be approached in a positive and
supportive manner through the provision of needed treatment
and immediate services, without the potentially lengthy and
traumatic intervention process.
The Family Builders Program uses a family-centered, strength-and
community-based approach. Caseworkers involve all family
members in establishing goals for change. They focus on family
strengths, rather than problems, and help families to identify and
access family and community resources to reduce reliance on
formal support systems, such as welfare. For example, rather
than focusing on a mother’s unemployment, a caseworker
would focus on the family’s access to a car as a resource that the
mother can use to seek and maintain employment. A wide range
of services is provided through local networks of community-based
agencies. By drawing from these local resources to meet
families’ needs, the program fosters interdependence between
the family and the community.
Family Builders’ premise
is that families with low
child abuse/neglect risk
require different responses
from CPS.
Introduction and Background
3
OFFICE OF THE AUDITOR GENERAL
Figure 1
Family Builders Program
Contractors by County
As of June 2001
Arizona Baptist Children’s Services, Arizona’s Children Association, Arizona Partnership for Children,
Black Family and Child Services, CareNow, and Child and Family Resources
Arizona Partnership for Children
Arizona Partnership for Children, Marana Unified School District, and Our Town Family Center
Child and Family Resources
Counties not covered by the program
Source: Auditor General staff analysis of the Department of Economic Security’s Family Builders Request
for Proposals and contracts for fiscal year 2000.
Mohave
La Paz
Yuma
Pima
Santa
Cruz
Maricopa
Pinal
Cochise
Graham
Greenlee
Gila
Coconino
Yavapai
Navajo
Apache
Introduction and Background
4
OFFICE OF THE AUDITOR GENERAL
Arizona’s Family Builders Program has been nationally recog-nized.
In March 2001, it placed as a semifinalist for the 2001 In-novations
in American Government Award granted by the John
F. Kennedy School of Government at Harvard University.
Family Builders:
How It Operates
Family Builders operates within a CPS referral system that moni-tors
child abuse and neglect reports received by the CPS hotline
(see Figure 2, page 5). Hotline workers prioritize each report into
one of four categories, ranging from potential- to high-risk. Both
potential- and low-risk reports are referred to a Family Builders
district coordinator at CPS who decides whether the report
should remain within CPS for investigation or be assigned to a
Family Builders contractor. In order to meet legislative require-ments
for child safety, certain reports, such as those with allega-tions
of sexual abuse, are ineligible for Family Builders, regard-less
of their priority. If the report is eligible, the district coordina-tor
refers the reports to a contracted service provider in the fam-ily’s
area. When the contracted service provider receives the re-ferral,
a caseworker is assigned to the family. The caseworker
visits the family’s home within 48 hours of receipt of the referral,
excluding weekends. The caseworker informs the family that
CPS received a report about them and referred it to Family
Builders and offers services.
If the family agrees to participate, the caseworker and family
complete an assessment to identify the family’s strengths and
prioritize their needs. In the third and final program phase, the
service plan, the family participates in developing a service plan
based on its strengths and resources. During this phase, the fam-ily
receives various services, such as counseling and vouchers for
clothing and food, to address the family’s circumstances that re-sulted
in the child abuse report. The service plan phase is an on-going
process. For example, a family who manages to avoid evic-tion
through rental assistance from Family Builders may then set
a goal to move to less-expensive housing. Throughout the ser-vice
plan phase, the caseworker conducts regular home visits to
monitor the family’s progress and assess the children’s safety.
The program targets fami-lies
with reports of low or
potential risk of abuse or
neglect.
Introduction and Background
5
OFFICE OF THE AUDITOR GENERAL
Figure 2
Family Builders Program Process
As of June 2001
Source: Auditor General staff analysis of Family Builders policy and procedures training material.
Option 3
Option 1
Option 2
CPS District Coordinator
verifies Family Builders eli-gibility
and forwards report
to Family Builders contrac-tor.
Family Builders case-worker
attempts 2 unan-nounced
visits to the fam-ily
within 48 hours (ex-cluding
weekends) to offer
voluntary services.
If the case worker ob-serves
signs of abuse or
neglect, the case is re-ferred
back to CPS.
If family declines ser-vices,
the report is
closed and CPS is in-formed
of case closure.
If family accepts services,
the Family Builders
caseworker performs an
assessment.
Family Builders caseworker and
family create a family-centered
service plan. Family chooses
which services they want to re-ceive.
If the caseworker observes
signs of abuse or neglect
during service provision, the
caseworker makes a new
Case is closed when family report to the CPS hotline.
completes service plan or no
longer desires services.
CPS hotline receives a
report of low or poten-tial
risk for child abuse
or neglect.
Family
Builders
Report
Closed
Family
Builders
Report
Closed
Introduction and Background
6
OFFICE OF THE AUDITOR GENERAL
The program’s intent is to close the case when a family completes
their service plan goals. However, a family may end contact with
the program during any phase for any reason. For example, a
family may move out of state or simply decline to participate fur-ther.
By statute, Family Builders offers, at a minimum, the following
services:
¾ Family assessments
¾ Case management
¾ Parenting skills training
¾ Parent aide services
¾ Housing search and relocation
¾ Emergency services, such as food and rental assistance
¾ Supportive intervention/guidance counseling
¾ Intensive family preservation services, such as crisis counsel-ing
¾ Child daycare
¾ Transportation
¾ Respite services
¾ Shelter service
Families choose which services they want to receive; therefore,
not all families receive each of the above services. Typically,
families participate in the program for about 6 months and re-ceive
case management, counseling, and financial assistance,
such as a rent or utilities payment.
Contractors are required to establish local advisory boards that
include representatives from local community agencies and
Introduction and Background
7
OFFICE OF THE AUDITOR GENERAL
families. The boards are intended to help Family Builders con-tractors
provide oversight to the program and strengthen their
resource network by identifying service providers and enlisting
their support.
Appropriations and Contracting
The Legislature appropriated approximately $9.5 million to the
Department for Family Builders in fiscal year 2000. Approxi-mately
$8 million was appropriated each year for fiscal years
2001 and 2002. These amounts include both General Fund and
Temporary Assistance for Needy Families (TANF) appropria-tions
(see Table 1, page 8). Although the overall appropriation
for the program has remained constant the last 2 years, the De-partment
has received TANF monies in place of most of its Gen-eral
Fund monies.
In addition to funding services for families, the appropriations
pay for two full-time Family Builders Program specialists who
are responsible for administrative tasks, including training con-tractor
staff and monitoring service provision and data collection
on a quarterly basis. The specialists conduct quarterly Statewide
Partnership meetings with contractors to discuss program up-dates
and policy changes. The appropriations also fund one full-time
contract specialist who monitors program contracts.
There are 8 Family Builders contractors that provide services to
families in 10 of Arizona’s 15 counties. Contractors are paid a
capitated rate, a set amount for each of the three program phases,
which they use to provide services to clients in collaboration
with local community agencies participating on a contractual,
fee-for-service, or in-kind basis.
Family Builders received 14,249 referrals between August 1, 1999
and April 30, 2001. As of April 30, 2001, in 9,586 cases (67 per-cent),
the family had completed only the referral phase. In 4,663
cases (33 percent), the family received an assessment. In all but
266 of the assessed cases, the family signed a service plan. Table 2
(see page 9) shows for each contractor the total number of
Introduction and Background
8
OFFICE OF THE AUDITOR GENERAL
Table 1
Family Builders Program
Statement of Revenues and Expenditures
Years Ended or Ending June 30, 2000, 2001, and 2002
(Unaudited)
2000
(Actual)
2001
(Actual)
2002
(Estimated)
Revenues:
Appropriations: 1
State General Fund $6,573,700 $2,771,600 $1,804,300
Temporary Assistance for Needy Families (TANF) 2,922,400 5,227,400 6,200,000
Total Revenues $9,496,100 $7,999,000 $8,004,300
Expenditures and other uses:
Personal services and employee-related 182,093 208,839 210,000
Travel, in-state 1,082 86 100
Aid to individuals 2 9,226,031 7,782,445 7,786,550
Other operating 1,655 7,630 7,650
Total expenditures 9,410,861 7,999,000 8,004,300
Reversions to the State General Fund 3 85,239     
Total expenditures and reversions to the State
General Fund $9,496,100 $7,999,000 $8,004,300
1 Amounts are the final appropriations for the Family Builders Program, including all supplemental
appropriations and authorized appropriation transfers.
2 Amounts are payments made, or estimated to be made, to service providers. The 2001 amount includes $46,877
obligated to providers, but not paid at August 13, 2001.
3 Department-estimated reversions to the State General Fund for 2000. Since lapsing appropriations are not closed
until 2 years after the end of the fiscal year, actual reversions were not available at the time of this report.
Source: Auditor General staff analysis of the State of Arizona Appropriations Report and various reports of the
Arizona Department of Economic Security Financial Management Control System for the years ended
June 30, 2000 and 2001, and Division estimates of financial activity for the year ended June 30, 2002, as of
August 31, 2001.
Introduction and Background
9
OFFICE OF THE AUDITOR GENERAL
Family Builders referrals received, and the percentage of cases
with assessments and service plans. In developing the program
budget, the Department anticipated that at least 34 percent of the
cases referred to contractors would progress to the service plan
phase. As of April 30, 2001, only two contractors, Arizona Part-nership
for Children and CareNow, met or exceeded the ex-pected
rate of families who signed service plans (see Table 2).
Overall, the total percentage of cases in which a family signed a
service plan was below the Department’s expected rate. How-ever,
the percentage is higher than the percentage reported in
the previous evaluation (28 percent) (see Report No. 00-4). Fur-ther,
more contractors and the program as a whole may yet meet
or exceed the expectation since some of the cases remain open
and families may progress to higher stages of the program.
Table 2
Family Builders Program
Contractors, Number of Referrals, and Percentage of Cases
With Assessments and Service Plans
August 1, 1999 through April 30, 2001
Contractor Referrals1 Assessment
Service
Plan
Arizona Baptist Children’s Services 3,170 28% 27%
Arizona’s Children Association 1,575 25 24
Arizona Partnership for Children (2 contracts) 4,646 39 37
Black Family and Child Services 910 33 32
CareNow 1,372 38 34
Child and Family Resources (2 contracts) 957 33 31
Marana Unified School District 258 22 22
Our Town Family Center 1,356 28 24
Total 14,244 2 33% 31%
1 Includes open and closed cases.
2 Total referrals for the period were 14,249; however, because of missing contractor data, 5 referrals
were omitted from this analysis.
Source: Auditor General staff analysis of Family Builders data for cases referred between August
1, 1999 and April 30, 2001.
Introduction and Background
10
OFFICE OF THE AUDITOR GENERAL
Scope and Methodology
This evaluation focused on program effectiveness and admini-stration
and presents findings and recommendations in two ar-eas:
¾ Although services provided to families differ, Family Build-ers
and CPS had similar outcomes.
¾ The Department should take additional steps to improve
program monitoring and oversight.
Evaluators used multiple methods in the evaluation, including:
¾ Analysis of Family Builders Participants—Evaluators ana-lyzed
Family Builders program data to describe the demo-graphic
characteristics of Family Builders families who
signed a service plan and whose referring CPS report date
was between August 1, 1999 and April 30, 2001. This period
was selected to follow up from the previous evaluation,
which reported on families whose referring CPS report date
was between January 1, 1998 and July 31, 1999.
¾ Analysis of Subsequent CPS Reports—To determine the
program’s effectiveness in preventing subsequent child
abuse and neglect, evaluators collected and analyzed CPS in-volvement
data on families investigated by CPS and families
in Family Builders.
Ø Families investigated by CPS—Families investigated
by CPS were eligible for Family Builders but not referred
because the program was not available in the counties
where they live. This group includes families who live in
Gila, La Paz, Mohave, Pinal, or Yuma County, where the
Family Builders program is unavailable (see Figure 1,
page 3) and whose referring CPS report date was be-tween
January 1, 1998 and September 30, 2000. The group
also includes families who live in Apache, Cochise, Co-conino,
Graham, Greenlee, Navajo, Santa Cruz, or Yava-pai
County and whose referring CPS report date was be-tween
January 1, 1998 and May 30, 1999, during which
time Family Builders was unavailable in those counties.
Introduction and Background
11
OFFICE OF THE AUDITOR GENERAL
Ø Families in Family Builders—Family Builders families
were divided into three groups: those who signed and
completed a service plan, those who signed and did not
complete a service plan, and those who were assessed
only. These families had a referring CPS report date be-tween
January 1, 1998 and September 30, 2000, and a clo-sure
date on or before September 30, 2000. The time pe-riod
was selected to capture subsequent CPS reports, if
any, for a minimum of 6 months following families’ in-volvement
with the program.
Evaluators compared the Family Builders families’ and CPS
families’ rates of subsequent CPS reports and analyzed whether
the reports were substantiated, unsubstantiated, or referred to
Family Builders.
n Analysis of Statewide CPS Report Rates—Evaluators ex-amined
statewide rates of CPS reports to determine whether
there was a difference over time in the report rates in areas
served by Family Builders and areas served only by CPS.
n Risk Data Analysis—To determine the program’s effective-ness
in reducing the risk for child abuse and neglect, evalua-tors
analyzed data on Family Builders families that either
signed and completed a service plan or signed and did not
complete a service plan. Risk data came from two assessment
tools: the Family Risk Scales and the Brief Family Assessment
Scale (see Appendix, page a-i through a-vii). Evaluators
compared the families’ risk at program entrance to their risk
at program exit to assess change due to program participa-tion.
n Database Validation—To verify the accuracy and complete-ness
of the cost and participant information contained in the
Family Builders database, evaluators analyzed 100 randomly
selected case files in 2 separate reviews, and compared the
files to the Family Builders database. They also corrected the
identified errors, where possible, and otherwise excluded
fields and cases to prepare the data for analysis.
Introduction and Background
12
OFFICE OF THE AUDITOR GENERAL
n Document Review—Evaluators reviewed Family Builders
program contracts and brochures to describe the participat-ing
community agencies, the geographical areas served, and
the services provided. Evaluators also reviewed the Family
Builders program manual and training materials to assess the
technical assistance provided by the Department. To evaluate
the Department’s monitoring, evaluators reviewed moni-tored
case files and compared Family Builders Specialists’
monitoring reports to contractors’ responses, known as cor-rection
plans. Finally, evaluators examined membership lists
and meeting minutes from each contractor’s local advisory
board meetings to assess their compliance with legislative re-quirements.
n Site Observation—Evaluation staff visited each Family
Builders contractor and accompanied staff on home visits to
program clients to observe the program as implemented. To
assess compliance with legislative requirements, evaluators
also attended local advisory board meetings and meetings
between participating community agencies, known as col-laborators’
meetings.
Acknowledgements
The Auditor General and staff express appreciation to the direc-tor
of the Department of Economic Security, the staff of the De-partment’s
Division of Children and Family Services, and the
Family Builders program supervisors and staff for their coopera-tion
and assistance throughout the evaluation.
13
OFFICE OF THE AUDITOR GENERAL
FINDING I SERVICES DIFFER BUT
PROGRAM OUTCOMES
COMPARABLE TO CPS
Although the services provided to families differ, Family Build-ers
and CPS had similar outcomes related to subsequent CPS re-ports.
Family Builders offers more services to low-risk families
than CPS does, yet the proportion of families served by Family
Builders who received subsequent CPS reports was comparable
to the proportion of families investigated by CPS who received
subsequent reports. However, families completing the program
did have fewer subsequent reports than families who did not
complete the program. Furthermore, participants reduced their
risk for abuse and neglect.
Family Builders and
CPS Had Similar Outcomes
While Family Builders provides more services to low-risk fami-lies
than CPS provides, many program participants received
subsequent CPS reports. The proportion of program families
who received subsequent reports was similar to the proportion
of families served by CPS who received subsequent reports. In
addition, CPS report rates over time were similar in areas of the
State served by Family Builders and areas served only by CPS.
Program families receive more services—Family Builders offers a
larger scope of services to families than CPS does. For instance,
although Family Builders could provide a family with car repair
services, CPS could not. Its services are limited to those within
CPS and the Department, many of which are available only to
families whose child has already been removed from the home.
Furthermore, the CPS intervention is not characterized by a fam-ily-
centered, strength-based approach, as Family Builders is.
Finding I
14
OFFICE OF THE AUDITOR GENERAL
Finally, while all Family Builders families receive case manage-ment
throughout the program, CPS families generally receive
case management only after an investigation.
Many participating families received subsequent reports—
Although they received more services, 40 percent of program
families received subsequent CPS reports, a greater percentage
than in the previous evaluation. Families served by the program
include families who signed and completed a service plan and
families who signed but did not complete a service plan. Typi-cally,
these families received 2 subsequent reports within 22
months of the referring report. Subsequent reports could be sub-stantiated,
unsubstantiated, or referred to Family Builders.
Of the 3,135 families served by the program, 8.9 percent received
at least 1 substantiated subsequent report, 27.5 percent received
at least 1 unsubstantiated subsequent report, and 17.6 percent
received at least 1 subsequent report that was referred to Family
Builders.1 The proportion of families with subsequent reports is
greater than the proportion that evaluators found in the last
evaluation (see Report No. 00-4). Previously, evaluators found
that 32 percent of families served by the program received sub-sequent
reports.2
1 Because some families receive multiple reports, each of which can be sub-stantiated,
unsubstantiated, or referred to Family Builders, they may be
counted in more than one category. Therefore, the percentages cannot be
summed.
2 The current reporting period is longer and contains many of the same
families analyzed in the prior reporting period, plus many new families
who have since entered Family Builders.
Substantiated reports—represent cases in which an investigation
concluded that abuse or neglect had occurred.
Unsubstantiated reports—represent cases in which abuse or
neglect was alleged but not substantiated in an investigation.
Referrals to Family Builders—represent cases in which abuse or
neglect was alleged but services were offered in lieu of an
investigation.
Finding I
15
OFFICE OF THE AUDITOR GENERAL
Proportions of families with subsequent reports were compara-ble
in Family Builders and CPS—Family Builders participants
did not have fewer subsequent reports than the low-risk families
investigated by CPS. As Table 3 (see page 16) indicates, 34.8 per-cent
of the families investigated by CPS received subsequent CPS
reports, as compared to the 35.6 percent of families who com-pleted
the Family Builders program and received subsequent
reports. When examining only subsequent substantiated reports,
the proportions were lower for both groups. As indicated in the
table, 8.9 percent of families investigated by CPS received subse-quent
substantiated reports, as compared to 6.6 percent of fami-lies
who completed the program. This difference was statistically
significant. However, when analyzing the families investigated
by CPS separately by CPS district, there were no statistically sig-nificant
differences between families in three of the four districts
served by CPS compared to families completing the program.
Finally, among families with no prior CPS reports, 5.5 percent of
families who completed Family Builders had subsequent sub-stantiated
reports, whereas 8.1 percent of families investigated by
CPS had subsequent substantiated reports.1
1 There were 1,349 families that completed service plans and 5,931 families
investigated by CPS who had no prior CPS reports.
Seven percent of Family
Builders completers re-ceived
subsequent sub-stantiated
CPS reports, as
compared to 9 percent of
families investigated by
CPS.
Finding I
16
OFFICE OF THE AUDITOR GENERAL
Report rates were comparable in program areas and areas served
by CPS—Evaluators also compared the number of CPS reports
per 10,000 persons for areas served by Family Builders and areas
served only by CPS and found them to be similar. Although the
number of reports declined in 1998 after the inception of Family
Builders, this decline also occurred in areas where Family Build-ers
was not available and, therefore, could not be attributed to
Family Builders. Further, the rates of reports in all areas of the
State stabilized in 1999 and remained stable through the end of
Table 3
Family Builders Program
Families with Subsequent CPS Reports
Family Builders Participants vs. Families Investigated by CPS1
January 1, 1998 through March 31, 2001
Group
Percentage of Families
With Subsequent Reports
Families referred to Family Builders
Substantiated
Reports3
Total
Reports2
Service plan completers (N=1,847) 6.6% 35.6%
Service plan noncompleters (N=1,288) 12.3 46.3
Families assessed only (N=369) 11.7 40.7
Families investigated by CPS (N=6,805) 8.9 34.8
1 Includes four CPS districts that did not have the Family Builders Program during the entire time period
from January 1, 1998 through March 31, 2001.
2 The difference between service plan completers and families investigated by CPS was not statistically
significant. The differences between the different groups of families referred to Family Builders were
statistically significant at the p<0.05 level. This means that the probability that the differences in
proportions were due to chance is less than 5 in 100.
3 The difference between service plan completers and families investigated by CPS was statistically
significant at the p<0.05 level. However, when comparing each of the four districts separately to service
plan completers, there was no statistically significant difference for three of the four districts. The
differences between the different groups of families referred to Family Builders were statistically
significant at the p<0.05 level.
Source: Auditor General staff analysis of Child Protective Services data from January 1, 1998 through
March 31, 2001.
Finding I
17
OFFICE OF THE AUDITOR GENERAL
the analysis period. The patterns of rates over time for Family
Builders and CPS areas are similar.
Positive Differences Exist
Among Program Participants
Although Family Builders and CPS had similar outcomes, posi-tive
differences exist among program participants. In particular,
families that completed the program had fewer subsequent re-ports
than families who did not complete the program. Further-more,
families receiving services experienced a slight reduction in
their risk for child abuse and neglect, as measured by the case-worker-
completed Family Risk Scale. The results from the family-completed
Brief Family Assessment Scale (BFAS) used to meas-ure
a family’s perception of stress and the resulting risk for child
abuse and neglect were inconclusive due to a low response rate.
Service plan completion resulted in fewer subsequent reports—
Although over two-thirds of the families referred to Family
Builders never accepted services, those families that completed
service plans received fewer subsequent CPS reports than the
other families. As Table 3 indicates (see page 16), 35.6 percent of
the families who signed and completed a service plan had sub-sequent
reports, as compared to 46.3 percent of the families who
signed but did not complete a service plan and 40.7 percent of
the families who were assessed but did not go on to sign a ser-vice
plan. Service plan completers also had fewer subsequent
substantiated reports (6.6 percent) than families who did not
complete their service plans (12.3 percent) and families who
were assessed only (11.7 percent). These differences may be at-tributed
to the length of time families spent in the program, the
number of services they received, or the families’ own motiva-tion
to complete the program or decrease their risk for abuse or
neglect.
Families receiving services reduced their risk for child abuse and
neglect—Although results from the BFAS were inconclusive, re-sults
from the Family Risk Scale indicate that program families
receiving services experienced a slight reduction in risk regard-less
of whether they completed their service plans. The Family
Risk Scale, a caseworker-completed assessment of the family’s
Finding I
18
OFFICE OF THE AUDITOR GENERAL
child-centered, parent-centered, and economic risk for child mal-treatment
(see Appendix, pages a-i through a-vii), was conducted
at program entrance and exit on 2,958, or 94 percent, of the 3,135
families served by Family Builders. Scores range from one to six
with one indicating low risk. Although families had low risk lev-els
at program entrance, their risk still declined slightly in each of
the three measured areas (see Table 4).
The results from the family-completed BFAS were inconclusive
due to a low response rate. The BFAS measures a family’s per-ception
of stress and the resulting risk for child abuse and neglect
(see Appendix, pages a-i through a-vii). Only 611, or 19 percent,
of the 3,135 families who signed service plans completed pre-and
post-tests. Therefore, although results from the BFAS indi-cate
that participants’ risk declined from program entrance to
exit, they cannot be used to assess the outcomes for the entire
program since not all families are represented. However, the
Department has made an effort to improve the response rate by
encouraging caseworkers to collect the data and by gathering
information on the reasons they do not collect the data. Conse-quently,
the percentage of completed assessments has more than
doubled since the Auditor General’s previous evaluation (see
Report No. 00-4).
Table 4
Family Builders Program
Family Risk Scale at Entrance and Exit
Scores of Families with Closed Service Plan Cases
January 1, 1998 through September 30, 2000
Average Risk Score
Risk Assessment Area Entrance Exit Change1
Child-centered risk 1.54 1.33 -0.21
Parent-centered risk 1.63 1.45 -0.18
Economic risk 1.62 1.36 -0.27
1 For all three risk areas, the reduction is statistically significant at the p<0.05 level.
This means that the probability that the change in risk scores occurred by chance
is less than 5 in 100.
Source: Auditor General staff analysis of pre- and post-scores on Family Risk
Scales as compiled in the Family Builders database.
19
OFFICE OF THE AUDITOR GENERAL
FINDING II DEPARTMENT HAS MADE
LIMITED PROGRESS IN
MONITORING AND OVERSIGHT
Although the Department has taken steps to address monitoring
and oversight problems identified in the Auditor General’s pre-vious
evaluation, its progress has been limited. The Department
still needs to improve its cost data in order to make statements
about the type and cost of services delivered. In addition, other
program data is often inaccurate or incomplete. Finally, contrac-tors
continue to need guidance in operating their local advisory
boards.
Improvement Still Needed
on Cost Data
The Department has attempted to address the problems with its
cost data that were identified in the Auditor General’s previous
evaluation. However, problems, such as missing and miscatego-rized
costs, persist. The data still need improvement if the De-partment
is to rely on them to make statements about the cost of
program services and the types of service most commonly pro-vided.
Department has taken steps to address identified problems—
The Department has taken steps to address problems identified
in the previous evaluation. The Auditor General’s March 2000
evaluation (see Report No. 00-4) found that the Department
lacked the necessary information to determine if contractors
were delivering adequate services and being paid at appropriate
rates. Specifically, it found that the Department’s reimbursement
rates were well above contractors’ costs, yet needed services
were not being provided. In addition, the evaluation also found
that the Department lacked complete and accurate information
on the type, number, and cost of services delivered.
Finding II
20
OFFICE OF THE AUDITOR GENERAL
Since the previous evaluation, the Department’s program spe-cialists
have begun to review contractors’ cost records during
their quarterly monitoring visits. In addition, the Department
convened a work group to review and revise cost categories and
their definitions. It also modified its reimbursement rates in its
fiscal year 2001 contracts to reduce the difference between the
rates and recorded costs.
Reliable cost data still not available—Despite the Department’s
efforts to improve cost data, problems persist. In a random sam-ple
of 100 cases, evaluators found many problems, such as miss-ing
and miscategorized costs. For example, the costs of services
provided during the referral or assessment phases were often
recorded as costs of the service plan phase. Some costs, such as
those for case management, were not recorded at all in some
cases. A primary cause of problems has been inadequate cost
category definitions, which allowed contractors to interpret the
categories in several ways and, consequently, to inconsistently
report costs across categories. Although the Department’s work
group recently revised definitions, the database still contains
data entered according to the old definitions. Without accurate
and complete cost information, the cost of program services and
the types of service most commonly provided still cannot be re-liably
determined.
Improvement Also Still
Needed on Other Program Data
The Department still needs to improve its other program data to
eliminate problems, such as missing and duplicative records,
and ensure its validity. Although several factors, including flaws
in the Family Builders database and inadequate training, system
monitoring, and technical support, have contributed to the per-sistence
of data problems, the Department has made an effort to
address them.
Problems remain in other program data—Evaluators identified
several problems in the other program data, including those pre-viously
identified in the last evaluation. In the same random
sample of 100 cases, evaluators found missing, inaccurate, and
duplicative program data, such as missing demographic and
Evaluators found prob-lems,
such as missing and
miscategorized costs.
Finding II
21
OFFICE OF THE AUDITOR GENERAL
risk assessment data, duplicate records, and cases with indistin-guishable
identifying information. For example, in numerous
instances, a single case was recorded in the database under mul-tiple
CPS report numbers or two different contractor names.
These findings are consistent with those in the Auditor General’s
previous evaluation, which found that the information collected
on program participants was incomplete and inaccurate.
Several factors explain the persistence of data problems—Data
problems persist due to several factors associated with the De-partment’s
data collection system. Specifically,
n Family Builders database needs redesign, user’s man-ual—
The Family Builders database contains many design
flaws. For example, the database inefficiently requires that
the same information, such as the report date, be entered
numerous times, potentially allowing for increased errors in
the data. Also, the database’s data entry screens do not match
the program’s forms, which can make data entry confusing
and allow for errors in interpretation. In addition, although
the 2000 evaluation recommended that the Department cre-ate
a database user’s manual, it has not done so.
n Training is inadequate—The Department does not offer
regular training focused on the database. Furthermore, exist-ing
training does not sufficiently address common data col-lection
and entry problems and frequent errors, and is not tai-lored
to the program staff’s differing needs.
n System monitoring is insufficient—The Department’s
monitoring does not include sufficient system-wide analysis
to address errors. For example, the Department does not ex-amine
whether the data is complete and consistent across ta-bles
in the database and across contractors. Existing monitor-ing
efforts focus on case-by-case errors and are, therefore,
unlikely to detect and address system-level errors and their
causes.
n Additional technical support is needed—Additional tech-nical
support is needed because the Department’s two pro-gram
specialists do not have the technical knowledge and
skills needed to address many of the problems, and a man-
Finding II
22
OFFICE OF THE AUDITOR GENERAL
agement analyst assigned to assist them is only available
part-time.
The Department has made an effort to address problems—The
Department has attempted to address data problems and elimi-nate
their causes. It established a task force, comprising the Fam-ily
Builders program specialists and contractors, to review data
collection forms and program definitions. The Department also
allowed the management analyst to temporarily work primarily
with Family Builders to reduce database design flaws, install a
newly revised Family Builders database at contractors’ sites, and
provide database training as part of a full-day workshop for
contractors. More recently, when the analyst position was
vacated, the Department hired a consultant to improve the
Family Builders database. In addition, the Department added
language about contractors’ data collection responsibilities to its
fiscal year 2002 contracts.
The Department should take additional steps to improve its cost
and program data—To improve the data collection system, the
Department should develop a data quality assurance plan detail-ing
measures that will allow for the ongoing review and im-provement
of the data collection system. It should also conduct a
system-wide review of the Family Builders database to eliminate
current problems, such as duplicate records and cases with mul-tiple
identification numbers, and modify the database to prevent
such problems in the future. To ensure that recently modified
program definitions and cost categories are interpreted correctly
and result in valid data, the Department should monitor their
use by contractors. In addition, the Department should create a
database users’ manual and reinforce its contents in trainings
held at regular intervals. Training should explain proper data
collection and verification procedures and be tailored to meet the
needs of the different audiences who handle program data:
caseworkers, data entry clerks, and program supervisors. Finally,
the Department should assign to the program the technical re-sources
required to effectively address current problems in the
short term and maintain the data system over the long term.
The Department estab-lished
a task force to re-view
data collection forms
and program definitions.
Finding II
23
OFFICE OF THE AUDITOR GENERAL
Contractors Continue To
Need Guidance in Operating
Local Advisory Boards
Despite the Department’s efforts to assist, contractors need addi-tional
guidance in operating their local advisory boards in accor-dance
with statute. The boards have ongoing membership and
productivity problems and are not adequately monitored by the
Department.
Local advisory boards still not fulfilling their responsibilities—
The Department has not made certain that local advisory boards
are meeting their statutory responsibility to ensure service avail-ability,
provide program direction, and represent the commu-nity.
As in the previous evaluation period, evaluators found that
the boards have problems meeting membership requirements
and providing guidance to contractors. To assist with member-ship
recruitment, the Department gave contractors lists of com-munity
organizations from which they might recruit board
members. Yet, contractors continue to have difficulty recruiting
board members. For example, although contractors are to in-clude
Family Builders families on the boards, three of the eight
contractors did not have family representatives as board mem-bers.
Furthermore, members’ attendance at meetings continues
to be inconsistent for most local boards.
Although the Department created a self-monitoring tool for con-tractors
to use to make board meetings more effective in address-ing
the boards’ statutory responsibilities, most boards continue
to have little impact on the program. Evaluators observed at least
1 advisory board meeting for each contractor and reviewed min-utes
from 25 board meetings. They found that although boards
generally provided information on local services and identified
some gaps in services, few boards set goals or assigned tasks re-lated
to addressing those gaps. Some contractors have indicated
that much of the information exchanged at board meetings is al-ready
exchanged among the collaborating agencies that serve
program families. This overlap and the difficulty in involving
board members has led some contractors to question the boards’
utility. Many contractors have indicated that they need more
guidance on how to run an advisory board.
Few local advisory boards
set goals or assigned tasks
to address service gaps.
Finding II
24
OFFICE OF THE AUDITOR GENERAL
The Department still does not adequately monitor local advi-sory
boards—Although the 2000 evaluation recommended that
the Department monitor board activities to ensure that legislative
requirements are met, the Department does not adequately do
so. For example, although the program specialists attend advi-sory
board meetings, they do not systematically review minutes
or membership lists or provide feedback on board activity to en-sure
that the boards fulfill their responsibilities.
Recommendations
1. The Department should develop a data quality assurance
plan for the ongoing review and improvement of the data
collection system.
2. The Department should conduct a system-wide review of the
Family Builders database to eliminate problems, such as du-plicate
entries, and modify it to prevent future problems.
3. The Department should monitor the use of the new program
definitions and cost categories, modifying them as needed, to
ensure that they are interpreted correctly and produce valid
data.
4. The Department should create a database users’ manual.
5. The Department should provide regular training on the data
collection system and tailor it to the needs of the different au-diences
who handle program data.
6. The Department should assign the program the technical re-sources
required to effectively address problems and main-tain
the data system.
7. The Department should monitor local advisory board activi-ties
to ensure that they meet their statutory responsibilities.
25
OFFICE OF THE AUDITOR GENERAL
STATUTORY EVALUATION COMPONENTS
Pursuant to Laws 2000, Ch. 23, §2, the Office of the Auditor Gen-eral
is required to include the following information in the Fam-ily
Builders Program evaluation.
B.1. Information on the number of reports referred to the
program.
Between August 1, 1999 and April 30, 2001, Family Build-ers
received 14,249 CPS reports concerning 31,892 chil-dren.
Seventy percent of the reports came from Maricopa
County. Twenty-one percent came from Pima County,
and 6 percent came from Apache, Coconino, Navajo, and
Yavapai Counties, collectively. The remaining 3 percent
came from Cochise, Graham, Greenlee, and Santa Cruz
Counties, collectively.
B.2. Information on the number of families who received
services.
Information on the number and characteristics of pro-gram
participants was available for families with CPS re-ports
dated August 1, 1999 through April 30, 001. Specifi-cally:
n 14,249 families were referred;
n 4,663 families were assessed; and
n Families signing service plans represented 11,117
children.
B.3. Demographic information on the families served.
Demographics are reported on families in the 4,397 cases
that signed a service plan to receive services.
Statutory Evaluation Components
26
OFFICE OF THE AUDITOR GENERAL
Figure 3
Family Builders Program
Primary Caregivers’ Marital Status
Divorced or
separated
(39%)
Widowed (3%)
Never married
(27%)
Married (31%)
Source: Auditor General staff analysis of Family Builders data on families who were
referred to the program and signed a service plan between August 1, 1999
and April 30, 2001.
n Caregivers—The majority of primary caregivers in
the program (93 percent) were female, and 7 percent
were male. In 95 percent of the cases, the caregivers
were the parents of the children. In almost all of the
remaining cases, other relatives cared for the children.
The average age of primary caregivers was 34. The
average age of female primary caregivers was 33, and
the average age of male primary caregivers was 39.
n Caregivers’ Marital Status and Living Arrange-ments—
Of the families served, fewer than one-third
of primary caregivers were married (see Figure 3).
Forty-eight percent of primary caregivers reported liv-ing
alone with their children, while 35 percent re-ported
living with a spouse or significant other.
Eleven percent of primary caregivers reported living
with parents or other relatives. The remaining 6 per-cent
lived with nonrelatives or in other arrangements.
Statutory Evaluation Components
27
OFFICE OF THE AUDITOR GENERAL
n Caregivers’ Education and Employment—Edu-cation
data was missing on 28 percent of the families
served. Of the cases for which education data was
available, most of the primary caregivers had a high
school education or higher (see Figure 4). Forty-six
percent of female primary caregivers and 68 percent
of male primary caregivers were employed either full-or
part-time.
n Caregivers’ Ethnicity—Ethnicity among primary
caregivers varied. Approximately 60 percent were
white, 26 percent were Hispanic, 7 percent were
Black, and 4 percent were American Indian. The re-maining
primary caregivers were either of mixed eth-nicity
or belonged to an ethnic group that is not
widely represented and, therefore, is categorized as
“other.”
n Family Size—One-quarter of the families served had
only one child. Twenty-nine percent had two chil-
Figure 4
Family Builders Program
Primary Caregivers’ Highest Level of Education Completed1
High School
(36%)
More than High
School (22%)
Less than High
School (42%)
1 Percentages were calculated on the 72 percent of families for whom education
data was available.
Source: Auditor General staff analysis of Family Builders data on
families who were referred to the program and signed a service
plan between August 1, 1999 and April 30, 2001.
Statutory Evaluation Components
28
OFFICE OF THE AUDITOR GENERAL
dren, 23 percent had three children, and the remain-ing
22 percent had four or more children.1
B.4. Type of services provided and families’ satisfaction
with the services.
n Services Delivered—All contractors are required to
offer 12 services: assessment, case management, child
care, housing search and relocation, parenting skills
training, supportive intervention and guidance coun-seling,
transportation, emergency, intensive family
preservation, parent aide, respite, and shelter services.
Because the Family Builders database contains miss-ing
and inaccurate information (see Finding II, pages
19 through 24), the specific type of services provided
for all cases cannot be determined. The Auditor Gen-eral’s
previous evaluation (Report No. 00-4) found
that all families received case management, 29 percent
received counseling services, 31 percent received par-enting
skills training or parent aide services, and 56
percent received emergency services, such as food or
clothing, or utilities, rent, and phone payments. An-ecdotal
data from the Department and Family Build-ers
contractors and evaluators’ observations from case
file reviews indicate that the services provided during
the current evaluation period were comparable to
those provided in the past.
n Client Satisfaction—Family Builders caseworkers
provided families with a confidential survey to meas-ure
their satisfaction with the services they received,
the program, and the caseworkers. Families were also
given a postage-free return envelope to mail the com-pleted
survey to the Office of the Auditor General.
The satisfaction survey was available in both English
and Spanish. Although 966 surveys were completed,
many had missing information about the client or the
contract. Consequently, only 676 of the surveys could
be linked to the 4,397 families who received services
during the evaluation period, making the response
1 Percentages do not total 100 due to rounding.
Statutory Evaluation Components
29
OFFICE OF THE AUDITOR GENERAL
rate 15 percent. The majority of 676 families whose
surveys were analyzed reported that they were satis-fied
with Family Builders. For example, 97 percent in-dicated
that the caseworkers treated their family and
their culture with a great deal of respect. Sixty-seven
percent said that they felt a great deal better able to
care for their children as a result of the program, while
23 percent said that they were somewhat better able to
care for their children. Fifty-nine percent reported that
the program met a great deal of their family’s impor-tant
needs, and 33 percent said that the program only
somewhat met their important needs. Finally, only 2
percent of the respondents reported that they were
not better off as a result of Family Builders.
B.5. The average cost of services provided.
As discussed in Finding II (see pages 19 through 24), the
Department still cannot produce reliable data on the cost
of services provided. Auditor General staff conducted a
review of Family Builders case files to verify data col-lected
in the Family Builders database. They found prob-lems
in the data, such as missing and miscategorized
costs. Because costs estimates were reported in the previ-ous
Family Builders evaluation (see Report No. 00-4),
evaluators decided to wait until the Family Builders da-tabase
was more reliable before developing new cost es-timates.
B.6. Information on providers.
There were eight Family Builders contractors, two of
which held two contracts each. All contractors provided
services in Pima and Maricopa Counties. In addition, two
contractors provided services in eight other counties, as il-lustrated
in Figure 1 (see page 3). Contractors provide ser-vices
to clients in collaboration with community health
and social service agencies that participate in the program
on a contractual, fee-for-service, or in-kind basis. Contrac-tors
generally provide case management services inter-nally
and provide other services, such as counseling, ex-
Statutory Evaluation Components
30
OFFICE OF THE AUDITOR GENERAL
ternally through community agencies. However, some
community agencies also provide case management to
Family Builders clients.
The Department anticipated that fewer than 65 percent of
the cases would progress beyond the referral phase, 1
percent would progress to, but not beyond, the assess-ment
phase, and that at least 34 percent of the cases re-ferred
to contractors would progress to the service plan
phase. As of April 30, 2001, only two contractors, Arizona
Partnership for Children and CareNow, met or exceeded
the expected rate of families who signed service plans
(see Table 2, page 9). However, more contractors may yet
meet the requirement since some of the cases remain
open and families may progress to higher stages of the
program.
n Arizona Baptist Children’s Association—Arizona
Baptist Children’s Association serves northwestern
Maricopa County. It works with 18 service providers,
including Blazing Saddles Equestrian Academy, Cal-vary
Rehabilitation Center, Christian Family Care
Center, Crisis Nursery, JobNet, The Bridge, and St.
Mary’s Food Bank. In addition, Arizona Baptist Chil-dren’s
Services employs 11 Family Builders case-workers
and 1 contract case manager to perform as-sessments
and provide services to their clients. Ari-zona
Baptist Children’s Services performed assess-ments
on 28 percent of all families referred to them
and provided service plans to 27 percent of all
families referred.
n Arizona’s Children Association—Arizona’s Chil-dren
Association serves southwestern Maricopa
County. Their four service providers are Florence
Crittenton, Southwest Human Development, Tum-bleweed,
and Southwest Behavioral Health. Five
caseworkers conduct assessments and provide ser-vices
to Family Builders clients. Arizona’s Children
Association performed assessments on 25 percent of
all families referred to them and provided service
plans to 24 percent of all families referred.
Statutory Evaluation Components
31
OFFICE OF THE AUDITOR GENERAL
n Arizona Partnership for Children (AZPaC)—
Arizona Partnership for Children holds two contracts.
The first contract represents service to families in
southeastern Maricopa County and northeastern
Pima County. Under this contract, their 22 participat-ing
service providers include Catholic Social Services
of Central and Northern Arizona, Devereaux Arizona,
Catholic Community Services, Open Inn, Parent’s
Anonymous, and Save the Family. Twenty full-time
(and one part-time) Family Builders caseworkers
conduct assessments and provide services to clients.
AZPaC’s second contract represents service to fami-lies
in Yavapai, Apache, Coconino, and Navajo Coun-ties.
Under this contract, the five participating service
providers include Arizona Baptist Children’s Services,
Arizona’s Children Association, Yavapai Catholic So-cial
Services of Central and Northern Arizona, Par-ent’s
Anonymous of Arizona, and Open Inn. Ten
caseworkers conduct assessments and provide ser-vices
to Family Builders clients. Through both con-tracts,
Arizona Partnership for Children performed
assessments on 39 percent of the families referred and
provided service plans to 37 percent of the families re-ferred.
n Black Family and Child Services, Inc.—Black Fam-ily
and Child Services serves families in south-central
Maricopa County. There are 14 participating service
providers, including Goodwill, YMCA, Sojourner
Center, Phoenix Day Child Development Center, and
Maricopa Skills Center. There are three Family Build-ers
caseworker positions. Black Family and Child Ser-vices
performed assessments on 33 percent of all fami-lies
referred to them and provided service plans to 32
percent of the families referred.
n CareNow—CareNow serves families in northeastern
Maricopa County. Four service agencies participate,
including Westside Social Services, Youth ETC., Valle
del Sol, and Jewish Family and Children’s Services. A
fifth agency, called Prehab, and its three caseworkers
withdrew from program participation during fiscal
Statutory Evaluation Components
32
OFFICE OF THE AUDITOR GENERAL
year 2000, leaving 13 caseworkers to conduct assess-ments
and provide services to Family Builders clients.
CareNow performed assessments on 38 percent of all
families referred to them and provided service plans
to 34 percent of the families referred.
n Child and Family Resources—Child and Family
Resources holds two contracts. The first contract
represents service to families in Central Phoenix. Un-der
this contract, there are 11 service providers, in-cluding
Maricopa County Parent Support Center, Cri-sis
Nursery, TERROS, and Mountain Park Health
Center. Four Family Builders caseworkers conduct as-sessments
and provide services to clients. Child and
Family Resources’ second contract represents service
to families in Graham, Greenlee, Cochise, and Santa
Cruz Counties. Under this contract they have three
participating service providers: Arizona Children’s
Association, Southeastern Arizona Behavioral Health
Services, and John Kelliher, PC. Currently, three
caseworkers conduct assessments and provide ser-vices
to Family Builders clients. Previously, Child and
Family Resources of Southeast Arizona had nine
caseworkers, but it reduced its case management staff
to adjust to fewer referrals in fiscal year 2000. One
part-time program supervisor also provides services
to clients. Through both contracts, Child and Family
Resources performed assessments on 33 percent of all
families referred to them and provided service plans
to 31 percent of the families referred.
n Marana Unified School District—Marana Unified
School District serves families in northwestern Pima
County. The one participating service provider is Ma-rana
Health Center. One caseworker and one pro-gram
supervisor conduct assessments and provide
services to Family Builders clients. Marana Unified
School District completed assessments for 23 percent
of the families referred and provided service plans to
22 percent of all families referred.
Statutory Evaluation Components
33
OFFICE OF THE AUDITOR GENERAL
n Our Town Family Center—Our Town Family Center
serves families in southeastern Pima County. It has
five participating service providers, including Ari-zona’s
Children Association, Casa de los Niños, Child
and Family Resources, and Jewish Family and Chil-dren’s
Services. Nine Family Builders caseworkers
conduct assessments and provide services to clients.
Our Town Family Center performed assessments on
28 percent of all families referred to them and pro-vided
service plans to 24 percent of the families re-ferred.
B.7. Recommendations regarding program administra-tion.
As discussed in Finding II (see pages 19 through 24), the
Department should improve its program oversight by:
1. Developing a data quality assurance plan for the on-going
review and improvement of the data collection
system and data validation;
2. Reviewing and modifying the Family Builders data-base
to eliminate current problems and prevent future
ones;
3. Monitoring the use of new program definitions and
cost categories and modifying them as needed to en-sure
the proper interpretation and collection of valid
data;
4. Creating a database user’s manual;
5. Providing regular training that is specific to data col-lection
and the Family Builders database, tailored to
the different needs of program staff, and emphasizes
consistency and accuracy in data collection;
6. Assigning the program the technical resources re-quired
to effectively address current problems and
maintain the data system; and
Statutory Evaluation Components
34
OFFICE OF THE AUDITOR GENERAL
7. Monitoring advisory board activities to ensure com-pliance
with legislative requirements.
B.8. Information concerning the extent to which the goals
and objectives of the program are being met.
The Department has one goal for the program: to enhance
parents’ ability to create safe, stable, and nurturing home
environments that promote the safety of all family mem-bers
and healthy child development. It established two
objectives: to increase the safety of children in their family
home and to increase parenting competence or effective-ness.
These objectives are associated with the following
yearly targets: fewer than 5 percent of families will be re-ferred
back to CPS with a substantiated report within 6
months of receiving services; and at least 99 percent of
families will reduce their risk scores in at least one risk
area. Only 3.1 percent of families served by the program
received subsequent substantiated reports within 6
months of the closure of their Family Builders case. How-ever,
the percentage of Family Builders families with sub-sequent
substantiated reports within 6 months was less
than 5 percent regardless of whether a family completed
the program. This indicates that a goal of less than 5 per-cent
may be more appropriate for the 6-month reporting
period. Furthermore, as the statistics in Finding I (see
page 13 through 18) indicate, over a longer period of time,
the proportion of families with subsequent substantiated
reports increases to more than 5 percent. Table 3 (see page
16) shows the proportion at 6.6 percent for families who
completed their service plan, 12.3 percent for families
who signed but did not complete their service plan, and
11.7 percent for families who were assessed only. There-fore,
it may be appropriate for the Department to adopt
an additional goal for a longer reporting period. On aver-age,
families receiving services slightly reduced their risk
scores on the Family Risk Scale in all three risk areas (see
Table 4, page 18), thereby meeting the program target.
Statutory Evaluation Components
35
OFFICE OF THE AUDITOR GENERAL
B.9. The outcomes for families served by the program,
including impact of the program in reducing the risk
of child maltreatment.
Finding I (see pages 13 through 18) provides information
on two outcomes for Family Builders families: subse-quent
CPS reports and risk for child abuse or neglect. The
outcome data indicate that there was no difference in ef-fectiveness
between Family Builders and CPS. Family
Builders families had comparable outcomes to those of
families investigated by CPS, although Family Builders
offers more services. Evaluators found no statistically
meaningful differences between the proportion of fami-lies
who completed Family Builders with subsequent re-ports
(35.6 percent) and the proportion of families inves-tigated
by CPS with subsequent reports (34.8 percent).
Furthermore, the statewide rates of CPS reports were
similar over time in areas served by the program and ar-eas
served only by CPS.
Although Family Builders and CPS have similar out-comes,
positive differences exist among program partici-pants.
In particular, program completers had fewer sub-sequent
reports than program dropouts, and the priority
of their subsequent reports was lower than that of the
dropouts. Furthermore, families served by Family Build-ers
experienced a slight improvement in their risk for
child abuse and neglect. Risk was measured by two as-sessment
tools; however, results were available for only
one tool. The caseworker-completed Family Risk Scale re-sults
show that risk scores declined in all three risk areas
from program entrance to exit. The results from the fam-ily-
completed BFAS were inconclusive due to a low re-sponse
rate, although the Department has taken steps to
increase completion of the assessment.
36
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
a-i
OFFICE OF THE AUDITOR GENERAL
APPENDIX
Assessment Tools
n Family Risk Scale—Based on their observations and inter-action
with the family, Family Builders caseworkers com-plete
the Family Risk Scale to assess the family’s risk for child
abuse and neglect. The Family Risk Scale is completed by the
caseworker at participants’ program intake and exit. The
Family Risk Scale emphasizes “parental characteristics and
family conditions that are believed to be predictors or precur-sors
of maltreatment.” 1
The Family Risk Scale is designed to measure risk in three
separate summary scales: child, parent, and economic risk.
The Family Risk Scale contains 26 individual dimensions that
assess the family’s adequacy and risk for the component be-ing
measured (see Table 5, page a-ii). Eight dimensions of the
Family Risk Scale apply to child-centered risk, 12 apply to
parent-centered risk, and 4 apply to economic-centered risk.
Three items are not categorized in any one risk area. The
caseworker provides a score for each dimension, using either
a 4-point, 5-point, or 6-point response scale. Within these
scales, “1” represents the highest degree of adequacy (and
lowest degree of risk) and higher numbers represent decreas-ing
degrees of adequacy. In addition, if a child in the family is
also a parent, caseworkers rate that child on two of the parent
dimensions.
1 Magura, Stephen, Beth Silverman Moses, and Mary Ann Jones. Assessing
Risk and Measuring Change in Families: The Family Risk Scales. Washington,
D.C.; Child Welfare League of America, 1987.
Appendix
a-ii
OFFICE OF THE AUDITOR GENERAL
Table 5
Family Builders Program
Family Risk Scale Assessment Areas and Scores
Risk Assessment Area
Range of
Scores1
Child-centered areas
Parent’s attitude to preventing placement 1-4
Emotional care and stimulation of child age 2 and older 1-5
Child’s mental health 1-5
Child’s school adjustment 1-6
Child’s delinquent behavior 1-6
Child’s home-related behavior 1-5
Child’s cooperation with agency (if also a parent) 1-4
Child’s preparation for parenthood (if also a parent) 1-4
Parent-centered areas
Adult relationships in household 1-4
Parent’s mental health 1-5
Knowledge of child care and development 1-4
Parent’s substance abuse 1-5
Parent’s motivation for problem solving 1-5
Parent’s cooperation with agency 1-4
Preparation for parenthood 1-4
Supervision of child under age 10 1-4
Parenting of child age 10 and over 1-4
Physical punishment of child 1-5
Verbal discipline of child 1-4
Emotional care and stimulation of infant under age 2 1-4
Economic risk factors
Habitability of family residence 1-5
Suitability of living conditions 1-5
Financial problems 1-4
Physical needs of child 1-4
1 For all three summary scales, a “1” represents the highest degree of adequacy (and lowest
degree of risk) and higher numbers represent decreasing degrees of adequacy.
Source: The Family Risk Scales developed by the Child Welfare League of America and
published by Stephen Magura, Beth Silverman, and Mary Ann Jones in Assessing Risk and
Measuring Change in Families: The Family Risk Scales, Washington, D.C.; Child Welfare
League of America, 1987.
Appendix
a-iii
OFFICE OF THE AUDITOR GENERAL
n Brief Family Assessment Scale (BFAS)—The BFAS is a
questionnaire completed by the family at program intake and
exit. The BFAS measures the family’s perception of their own
strengths, resources, and problems in the following seven ar-eas:
Personal Stress
Family Support
Economic Stress
Aggressive Behavior
Problems with My Child
Drug Use
Alcohol Use
The seven subscales comprise 76 individual statements. The
family is asked to indicate how often they experience the
situation described in the statement using the following scale:
1 None of the time
2 Very rarely
3 A little of the time
4 Some of the time
5 A good part of the time
6 Most of the time
7 All of the time
X Does not apply
For questions that apply to only one family member, families
are asked to respond based on the family member for whom
Appendix
a-iv
OFFICE OF THE AUDITOR GENERAL
the item most applies. For instance, if the person completing
the assessment never becomes hostile and threatening, but
another family member exhibits that behavior a good part of
the time, then the person should respond with a “5” to the
statement related to that issue.
Analysis of Subsequent
CPS Reports
To assess the impact of Family Builders on subsequent CPS re-ports,
evaluators analyzed CPS data from January 1, 1998 to
March 31, 2001, using several statistical methods. They examined
three groups of families referred to Family Builders: service plan
completers, service plan noncompleters, and families assessed
only. They also analyzed families investigated by CPS. They
used the Chi-Square test, the z test of proportional differences,
and analysis of variance to determine whether there were signifi-cant
differences between groups. They also analyzed the Phi sta-tistic
to assess the intensity of the association between variables
and determine whether any identified statistically significant dif-ferences
were meaningful and practical. Finally, evaluators con-ducted
multivariate regressions where appropriate to assess the
influence of factors, such as a family’s risk level, number of chil-dren,
and history of reports. Overall, they found no meaningful
differences between groups. Reported below are detailed results
from the more informative analyses.
n Analysis of total subsequent CPS reports—Evaluators
analyzed whether there were differences between groups in
the proportions of families with any subsequent CPS reports.
In a comparison of the four groups, the Chi-Square test of
significance yielded statistically significant differences (Chi-
Square=64.997, df=3, N=10,309, p=0.000). However, the in-tensity
of the relationship between participating in Family
Builders and receiving a subsequent CPS report was weak
(Phi = 0.079). Evaluators conducted the same analysis disag-gregating
the families investigated by CPS and analyzing
them by the CPS district in which they reside. Of the six CPS
districts in Arizona, four were not served by Family Builders
during all or some of the period of analysis: Districts 3, 4, 5,
and 6. See the Introduction and Background (page 3) for a
Appendix
a-v
OFFICE OF THE AUDITOR GENERAL
map showing coverage of the state by Family Builders. The
Chi-Square result of the district analysis yielded statistically
significant (119.908, df=6, N=10,309, p=0.000) but not mean-ingful
differences (Phi=0.108). Relative to the 35.6 percent of
families who completed Family Builders and had subsequent
CPS reports, 39.9 percent of District 3 families, 29.7 percent of
District 4 families, 34.7 percent of District 5 families, and 39.8
percent of District 6 families had subsequent reports.
The z test yielded no significant difference between families
who completed Family Builders and families investigated by
CPS (z=0.68, p=0.4966). However, the z test yielded statisti-cally
significant differences in the proportions of families
within Family Builders who received subsequent reports.
Specifically, families who completed a service plan had fewer
subsequent reports than families who signed but did not
complete a service plan (z=6.25, p=0.000) and families who
completed only an assessment (z=-1.90, p=0.0247).
n Analysis of subsequent substantiated CPS reports—
Evaluators analyzed the proportions of families who received
subsequent substantiated CPS reports. The Chi-Square test
revealed statistically significant (33.991, df=3, p=0.000) but
not statistically meaningful differences (Phi=0.057) between
the four groups. The z test yielded a significant difference be-tween
families investigated by CPS and families who com-pleted
service plans (z=-3.486, p0.000).
However, when the same analysis was conducted with fami-lies
investigated by CPS disaggregated by district, the z test
yielded statistically significant differences between families in
only one district, District 6, and Family Builders families
(z=7.27, p=0.000). The proportions of families investigated by
CPS in the other districts were not significantly different than
the proportions of families completing Family Builders. The
results for District 3 were z=1.33, p=0.1836. The results for
District 4 were z=-0.36, p=0.7188. The results for District 5
were z=1.16, p=0.246.
n Analysis of demographics characteristics—Evaluators
analyzed the primary caregiver’s age and sex and the fam-ily’s
number of children to identify differences between the
Appendix
a-vi
OFFICE OF THE AUDITOR GENERAL
groups of families. No statistically significant differences
were found between service plan completers and families in-vestigated
by CPS in the average primary caregiver’s age.
However, service plan completers had a higher proportion of
female primary caregivers (92.5 percent), relative to families
investigated by CPS (88.4 percent) (Chi-Square=24.314, df=1,
N=8,551, p=0.000). In addition, service plan completers had a
smaller proportion of white primary caregivers (62.3 percent)
than did families investigated by CPS (69.2 percent). No other
ethnic differences were found. No reliable data on number of
children was available for families investigated by CPS;
therefore, no comparison was made.
The average primary caregiver’s age in families who com-pleted
service plans (34.1) was statistically significantly
higher than the average primary caregiver’s age in families
who signed but did not complete service plans (32.1)
(p=0.00). However, there was no difference in average age
between service plan completers and families assessed only
or between noncompleters and families assessed only. No
statistically significant differences were found between the
Family Builders groups in the distribution of primary care-givers
by sex or ethnicity. Similarly, no differences were
found between groups in the average number of children in
the household. The median number of children for each
group was two.
n Analysis of families with no prior CPS reports—As re-ported
in Finding I (see pages 13 through 18), when looking
only at families whose first involvement with CPS during the
period examined was through the referring CPS report, the
analysis showed that there was a small difference between
the proportion of families served by Family Builders (29 per-cent)
and the proportion of families investigated by CPS (33
percent). The Chi-Square result was 204.396 (df=6, N=14,599),
p=0.00 and the Phi was 0.118. Again, there was a statistically
significant but weak association.
n Analysis of families referred to but not participating in
Family Builders—In addition to the groups described above,
two additional groups were included in the analysis of sub-sequent
reports. The first group included 5,265 families who
Appendix
a-vii
OFFICE OF THE AUDITOR GENERAL
were referred to Family Builders but declined to participate.
The second group included 2,333 families who were referred
to Family Builders but could not be located by the assigned
caseworkers. Families in these groups had a referring CPS
report date between January 1, 1998 and September 30, 2000,
and their cases were closed by September 30, 2000. Both
groups had a smaller proportion of families with subsequent
reports than the proportions of families served by Family
Builders and families investigated by CPS with subsequent
reports. The differences in proportions of families with sub-sequent
reports were statistically significant (Chi-Square=
229.535, df=6, N=17,907, p=0.000); however, the intensity of
the relationship was too weak (Phi = 0.113) to have meaning-ful
significance.
n Analysis of CPS report priority of originating report—
Potential- and low-risk reports are eligible for the Family
Builders program, but low-risk reports involve more serious
allegations of child abuse and neglect. The proportion of
families who completed Family Builders and had low-risk
originating reports (40 percent) was significantly smaller than
the proportion of families who were investigated by CPS and
had low-risk reports (58 percent). The Chi-Square result was
475.624 (df=18, N=17,907, p=0.00), indicating statistical sig-nificance.
However, the relationship between report priority
and program participation was weak (Phi=0.163).
Analysis of CPS Report Rates
Evaluators analyzed CPS data on all reports of child abuse and
neglect received in the state between January 1, 1998 and De-cember
31, 2000. The rates indicate the statewide number of CPS
reports made per month. Evaluators compared three sets of
rates: CPS Districts 1 and 2, served by Family Builders for the en-tire
analysis period; CPS Districts 3 and 6, served by Family
Builders beginning October 1, 1999; and Districts 4 and 5, served
only by CPS for the entire analysis period. They expected the rate
of reports in districts served by Family Builders to decline over
time and to be lower than the rate in districts served only by
CPS. However, the patterns of rates over time in all three areas
were similar.
a-viii
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
OFFICE OF THE AUDITOR GENERAL
Agency Response
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
__________________ ARIZONA DEPARTMENT OF ECONOMIC SECURITY __________________
1717 W. Jefferson, P.O. Box 6123, Phoenix, Arizona 85005
Jane Dee Hull John L. Clayton
Governor Director
Ms. Debbie Davenport, CPA
Office of the Auditor General
2910 North 44th Street, Suite 410
Phoenix, Arizona 85005
Dear Ms. Davenport:
The Department wishes to thank the Office of the Auditor General for the opportunity to
respond to the recently completed audit of the Family Builders Program.
The Department is pleased that the Auditor General has recognized the continued 100 percent
response rate to reports of child abuse and neglect utilizing a combination of investigations by
Child Protective Services and referrals to Family Builders. The report shows the goal and
objectives of the program were met by achieving the following outcome results. Ninety-seven
percent of program families who completed the program did not have a subsequent validated
report of child abuse and neglect within six months of receiving program services. The Family
Risk Scale assessment results indicate program families experienced a statistically significant
decrease in risk in each of the three measured areas: economic; parent-centered and child-centered.
An additional benefit of the program is families who completed the voluntary Brief Family
Assessment Scale showed a decline in risk from program entrance to exit. Participating families
are given a client satisfaction survey upon their exit from the program and 97 percent indicated
that the caseworkers treated their family and their culture with a great deal of respect. Only 2
percent of the respondents reported that they were not better off as a result of Family Builders.
The Department has made efforts to address monitoring and database problems and will
continue to do so. The Department convened a work group consisting of agency and
community provider personnel to address program issues. This group continues to meet and as
a result of their work, many changes to forms and the data base have been and will continue to
be made.
Page 2
Debbie Davenport
Once again, in March 2001, the Family Builders Program received national recognition. The
program placed as a semifinalist for the 2001 Innovations in American Government Award
granted by the John F. Kennedy School of Government at Harvard University.
The recommendations pertaining to each finding will be implemented as discussed in our
accompanying response. We would like to express our appreciation for the time and effort you
have invested in this important evaluation. We wish to specifically recognize Tanya Nieri for her
dedication during the evaluation process.
If you have any questions or need additional information please contact Anna Arnold, Assistant
Director of the Division of Children, Youth and Families at 602-542-3598, or me at 602-542-
5678.
Sincerely,
John L. Clayton
Enclosure
DEPARTMENT OF ECONOMIC SECURITY
RESPONSE TO THE
FAMILY BUILDERS PROGRAM EVALUATION
FINDING I: Services Differ But Program Outcomes Comparable to CPS
The Auditor General's report shows the goal and objectives of the program were met by
achieving the following outcome results. Just under 97 percent (96.9%) of program families did
not have a validated report of child abuse and neglect within six months of receiving program
services. The Family Risk Scale assessment results indicate that program families experienced a
statistically significant decrease in risk in each of the three measured areas: economic; parent-centered
and child-centered. Further, families who completed the voluntary Brief Family
Assessment Scale showed a decline in risk from program entrance to exit.
The Department agrees with the Auditor General's statement that families completing the Family
Builders program had fewer subsequent reports than families that did not complete the program.
The report further shows that these participants reduced their risk for child abuse and neglect.
During the period of January 1, 1998 through March 31, 2001, 8.9 percent of families
investigated by CPS received subsequent substantiated reports. Family Builders participants
who completed their service plan had a 6.6 percent substantiation rate during the same time
period. When rates of subsequent substantiated reports are looked at for program families
within a period of six months from the time they last received program services, the rate drops to
3.1 percent.
There are no recommendations under this finding.
DEPARTMENT OF ECONOMIC SECURITY
RESPONSE TO THE
FAMILY BUILDERS PROGRAM EVALUATION
FINDING II: Department Has Made Limited Progress In Monitoring and Oversight
The Department agrees with the Auditor General that steps to address problems identified in the
previous evaluation were made. The Department's Family Builders Program Specialists review
contractor's cost records during their quarterly monitoring visits and compare them to the
database for accuracy. The Department also modified the reimbursement rates paid to providers
based on the difference between the rates and recorded costs. In addition, the Department
convened a work group that has met multiple times to review and revise cost categories and
their definitions. This group has assisted in updating program forms and has offered suggestions
about changes to the database.
As programmatic issues are raised, the Department responds by addressing the concern in new
worker training, during the quarterly site monitoring visits of providers, and at the quarterly
partnership meetings of providers. The Department is committed to ensuring providers fulfill the
requirements of their contracts, so in turn, program families receive appropriate services. These
services are provided to families to assist them in creating a safe, stable and nurturing home
environment that promotes safety of all family members and healthy child development.
The finding of the Auditor General is agreed to and the following audit recommendations will be
implemented.
1. The Department should develop a data quality assurance plan for the ongoing review and
improvement of the data collection system.
The Office of Prevention and Family Support, Family Builders Program has been authorized to
begin a Family Builders Program Database Project, which will address the Family Builders Pilot
Program database issues. The Project will include a data quality assurance plan which will
address current problems and how to avoid future problems. The Project team will design a
process which will improve the data collection system and allow for ongoing data review. The
Project will begin with a review of the strengths and weaknesses of the current databases. The
Project team will complete a system-wide analysis to address errors and determine a means to
ensure completeness and consistency across tables and providers. The analysis will attempt to
eliminate the current database problems. If it is determined a new database is necessary, a
project team will be assembled to determine the design of the new database based on this
analysis. The stakeholders and users will be asked to provide guidance to the project team in the
development of the database and manuals. This
project team will be involved in the design, testing, and installation of the database. The team
will be involved in the development of the technical manuals required to support the database
and users. The project team, stakeholders and users will sign off on the final project and oversee
the installation of the database.
2. The Department should conduct a system-wide review of the Family Builders database to eliminate
problems, such as duplicate entries, and modify it to prevent future problems.
The Department recognizes the Family Builders Pilot Program Database has many design flaws.
The Project team will conduct a system-wide review of the Family Builders Pilot Program
Database. The team will correct as many errors as feasible in an effort to preserve the historical
data as accurately and completely as possible.
If it is determined a new database is needed, the system-wide review will provide the direction
for the design and development of a new Family Builders Program database that will contain
data which is complete, accurate and has integrity. The project may culminate in a historical
database and a new database supported by technical manuals.
3. The Department should monitor the use of the new program definitions and cost categories,
modifying them as needed, to ensure that they are interpreted correctly and produce valid data.
The Department convened a work group that has met multiple times to review and revise cost
categories and their definitions. The Department's Family Builders Program Specialists review
contractor's cost records during their quarterly monitoring visits and compare them to the
database for accuracy. To ensure the integrity of the cost data, the Department will design and
implement a more detailed monitoring process, which will focus on individual case errors as well
as system level error detection and their causes. The process will be used to monitor the
provider’s use and understanding of the current cost categories and program definitions. The
monitoring will include a systematic database review for errors and omissions. When monitoring
detects problems, training will be made available to address any problem the user may be
experiencing. This training will occur at regular intervals and on an as-needed basis. The
monitoring will be designed to ensure the program data is valid and that definitions and cost
categories are interpreted consistently.
4. The Department should create a database user’s manual.
The Project team will develop technical manuals, which will support the database and users.
The content of the manuals will be reinforced in regular ongoing training.
5. The Department should provide regular training on the data collection system and tailor it to the
needs of the different audiences who handle program data.
The Department's Family Builders Program Specialists have conducted programmatic training
multiple times since the inception of the program. Although the majority of the training focuses
on programmatic details, the importance of an accurate database is stressed. The Department
has provided technical support to providers on database issues and will continue to do so. To
further enhance the providers' ability to accurately collect and input data, the Project team will
develop additional training to support the database and meet the users' needs. The Department
will offer regular training focused on the database. The training will include proper data
collection and verification procedures. The training will be tailored to meet the needs of the
different audiences who handle the program data, such as caseworkers, data entry staff, and
program supervisors. The training will address common data collection and entry problems.
The training will focus on ensuring program data is accurate, complete, error-free and
appropriately categorized.
6. The Department should assign the program the technical resources required to effectively address
problems and maintain the data system.
The Department is committed to developing an effective database which is supported and
maintained by technically competent staff. Technical resources will be made available to
effectively address the current problems in the short-term and maintain the data collection
system over the long-term.
If a new database is needed, the Department’s goal is to retire the current database and launch
the new database with technical manuals by January 1, 2003.
The Project will be a concurrent effort, which will culminate in a historical database, and if
necessary, a new database, which is reliable, accurate, and complete.
The projected target dates include:
· Correct and preserve the historical information currently contained in the Family Builders
District Coordinators Database (Main) and the Family Builders Provider Database by
January 31, 2002;
· Review the needs and requirements of a new database and/or changes to be made to the
existing database to make it effective and recommend project approval to the executive
management team by April 1, 2002;
· Design the database and technical manuals based on user need and requirements by
September 1, 2002;
· Test the database by October 30, 2002;
· Provide training to the users by November 30, 2002;
· Install the new database or updates to the existing database to the users by December 15,
2002;
· Follow-up and provide technical assistance by January 1, 2003.
7. The Department should monitor local advisory board activities to ensure that they meet their
statutory responsibilities.
The Department created a self-monitoring tool for providers to use to make advisory board
meetings more effective in addressing the boards' statutory responsibilities. To assist with
membership recruitment, the Department gave providers lists of community organizations from
which they might recruit board members. The Department will continue to monitor local
advisory boards by providing training, attending meetings, and giving feedback. The providers
will again be trained on their statutory responsibility as it relates to the function of the advisory
board. Meeting notes and membership lists will be reviewed by Program Specialists and when
gaps in services are identified, Specialists will work with the provider to ensure they have a plan
on how these gaps will be addressed. In addition, the Department will give providers guidance
on how to engage and encourage family members to become involved. Specialists will also
provide guidance on how to encourage current board members to remain involved and attend
meetings on a consistent basis.
Other Performance Audit Reports Issued Within
the Last 12 Months
01-10
Future Performance Audit Reports
Perinatal Substance Abuse Program
Homeless Youth Intervention Program
01-1 Department of Economic Security—
Child Support Enforcement
01-2 Department of Economic Security—
Healthy Families Program
01-3 Arizona Department of Public
Safety—Drug Abuse Resistance
Education (D.A.R.E.) Program
01-4 Arizona Department of
Corrections—Human Resources
Management
01-5 Arizona Department of Public
Safety—Telecommunications
Bureau
01-6 Board of Osteopathic Examiners in
Medicine and Surgery
01-7 Arizona Department
of Corrections—Support Services
01-8 Arizona Game and Fish Commission
and Department—Wildlife
Management Program
01-9 Arizona Game and Fish
Commission—Heritage Fund
01-10 Department of Public Safety—
Licensing Bureau
01-11 Arizona Commission on the Arts
01-12 Board of Chiropractic Examiners
01-13 Arizona Department of
Corrections—Private Prisons
01-14 Arizona Automobile Theft
Authority
01-15 Department of Real Estate
01-16 Department of Veterans’ Services
Arizona State Veteran Home,
Veterans’ Conservatorship/
Guardianship Program, and
Veterans’ Services Program
01-17 Arizona Board of Dispensing
Opticians
01-18 Arizona Department of Correct-ions—
Administrative Services
and Information Technology
01-19 Arizona Department of Education—
Early Childhood Block Grant
01-20 Department of Public Safety—
Highway Patrol
01-21 Board of Nursing
01-22 Department of Public Safety—
Criminal Investigations Division
01-23 Department of Building and
Fire Safety
01-24 Arizona Veterans’ Service
Advisory Commission
01-25 Department of Corrections—
Arizona Correctional Industries
01-26 Department of Corrections—
Sunset Factors
01-27 Board of Regents
01-28 Department of Public Safety—
Criminal Information Services
Bureau, Access Integrity Unit, and
Fingerprint Identification Bureau
01-29 Department of Public Safety—
Sunset Factors