Evaluation, Family Builders Pilot Program

              State of Arizona
Office
of the
Auditor General
PERFORMANCE AUDIT
Report to the Arizona Legislature
By Debra K. Davenport
Auditor General
March 2000
Report No. 00-4
FAMILY BUILDERS
PILOT PROGRAM
The Auditor General is appointed by the Joint Legislative Audit Committee, a bipartisan committee
composed of five senators and five representatives. His mission is to provide independent and impar-tial
information and specific recommendations to improve the operations of state and local government
entities. To this end, he provides financial audits and accounting services to the state and political
subdivisions and performance audits of state agencies and the programs they administer.
The Joint Legislative Audit Committee
Representative Roberta L. Voss, Chairman
Senator Tom Smith, Vice-Chairman
Representative Robert Burns Senator Keith Bee
Representative Ken Cheuvront Senator Herb Guenther
Representative Andy Nichols Senator Darden Hamilton
Representative Barry Wong Senator Pete Rios
Representative Jeff Groscost Senator Brenda Burns
(ex-officio) (ex-officio)
Audit Staff
Carol Cullen—Manager
and Contact Person (602) 553-0333
Holly Whitt—Audit Senior
Copies of the Auditor General’s reports are free.
You may request them by contacting us at:
Office of the Auditor General
2910 N. 44th Street, Suite 410
Phoenix, AZ 85018
(602) 553-0333
Additionally, many of our reports can be found in electronic format at:
www.auditorgen.state.az.us
2910 NORTH 44th STREET • SUITE 410 • PHOENIX, ARIZONA 85018 • (602) 553-0333 • FAX (602) 553-0051
DEBRA K. DAVENPORT, CPA
AUDITOR GENERAL
STATE OF ARIZONA
OFFICE OF THE
AUDITOR GENERAL
March 21, 2000
Members of the Legislature
The Honorable Jane Dee Hull, Governor
Mr. John Clayton, Director
Department of Economic Security
Transmitted herewith is a report of the Auditor General, an evaluation of the Family
Builders Pilot Program. This evaluation was conducted pursuant to the provisions of
Laws 1997, Ch. 223, §3. I am also transmitting a copy of the Report Highlights to
provide a quick summary for your convenience.
As outlined in its response, although the agency does not agree with all of the findings,
it has agreed to implement all of the recommendations.
My staff and I will be pleased to discuss or clarify items in the report.
This report will be released to the public on March 22, 2000.
Sincerely,
Debbie Davenport
Auditor General
Enclosure
i
OFFICE OF THE AUDITOR GENERAL
SUMMARY
The Office of the Auditor General has completed an evaluation of
the Family Builders Pilot Program. This evaluation was conducted
pursuant to the provisions of Laws 1997, Ch. 223, §3, and provides
information regarding the Pilot Program’s implementation and ef-fectiveness
in achieving its goals.
The goal of the Family Builders Pilot Program is to enhance parents’
ability to create safe, stable, and nurturing home environments that
promote safety of all family members and healthy child develop-ment.
It is accomplished by responding to potential- and low-risk
reports of child abuse and neglect using a network of community-based
providers. Specifically, the Child Protective Services (CPS)
Division of the Department of Economic Security refers potential-and
low-risk reports of child abuse and neglect to community-based
Family Builders providers. Providers offer services aimed at reduc-ing
the problems contributing to the potential for abuse or neglect,
such as parenting skills training, counseling, housing search and
relocation, transportation, childcare, and emergency services. Par-ticipation
in the program is voluntary. Most families participate for
four to six months.
The Arizona Department of Economic Security is responsible for
administering the Family Builders Pilot Program. To do this, DES
contracts with eight community-based social service agencies in
Maricopa and Pima Counties to provide services to families. Provid-ers
are paid a capitated rate for completion at each of three service
delivery stages: referral, assessment, and service plan.
A total of 8,335 families were referred to Family Builders providers
from January 1, 1998 through July 31, 1999. Of these referrals, 5,578
families declined services; 2,757 families received an assessment; and
2,326 families continued on to complete a service plan and receive
services.
CPS refers potential- and
low-risk reports to Family
Builders.
Summary
ii
OFFICE OF THE AUDITOR GENERAL
Family Builders Unable to Show
Impact on Child Safety
at This Time
(See pages 13 through 17)
At this time, the Family Builders Pilot Program does not show an
effect on increasing children’s safety in their homes. As shown in
Table 1, there was no meaningful difference between the proportion
of Pilot Program families who received CPS reports alleging child
abuse or neglect subsequent to their involvement in the program
and the proportion of comparison group families who later received
CPS reports. One comparison group is comprised of families who
were referred to, but did not receive, Family Builders services, for
reasons such as they refused services, did not need services, could
not be located, or were referred back to CPS. The other group con-sists
of families who received DES investigations because the pro-gram
was not available in the county where they live. In addition,
Table 1
Family Builders Pilot Program
Comparison of Percentage of Families with Subsequent CPS Reports
Family Builders Participants vs. Comparison Group Families
January 1, 1998 through June 30, 1999
Percentage of Families with
Subsequent Reports1
Group
All
CPS Reports2
Substantiated
CPS Reports Only
Family Builders families 31.9% 6.5%
Families referred to, but not receiving, Family
Builders services3
24.2 5.7
Families outside Family Builders service area 28.5 6.6
1 Statistical tests were conducted to determine whether there was a meaningful difference in the proportion of Family
Builders participants compared to the proportion of comparison group families with subsequent CPS reports of child
abuse and neglect. Those tests are described in the Appendix, pages a-iii through a-v, and suggest that any observed di f-ferences
are due to chance.
2 Includes alleged and substantiated CPS reports.
3 These families were referred to the Family Builders program, but did not sign a service plan because they either refused
services, did not need services, could not be located, or were referred to CPS. Some of these families did participate in an
assessment.
Source: Office of the Auditor General staff analysis of Child Protective Services data from January 1, 1998 through June 30,
1999.
Summary
iii
OFFICE OF THE AUDITOR GENERAL
there was no difference in the proportion of families who received
subsequent reports where CPS substantiated that instances of abuse
or neglect occurred. Furthermore, incomplete or inaccurate informa-tion
about families served by the program limits the ability to ex-plain
the underlying reasons for these results. However, families
who accepted services from the Pilot Program experienced a slight
but statistically significant decrease in the risk for child abuse and
neglect, as measured by one instrument, the Family Risk Scale, com-pleted
by the Family Builders specialist. Another scale measuring
families’ perception of their risk for abuse or neglect could not be
analyzed because the providers did not collect enough self-assessments
from Pilot Program families.
DES Needs to Examine Costs
and Number of
Services Delivered
(See pages 19 through 27)
DES currently lacks the necessary information to determine if pro-viders
are delivering adequate services and are being paid at appro-priate
rates. There are indications that the rates paid to many pro-viders
may be higher than the costs of the services provided. A re-view
of 80 randomly selected cases found that the capitated pay-ment
rate exceeds providers’ actual costs by an average of $1,700 per
family served. There are also indications that clients may not be
receiving many of the services identified as necessary to meet their
needs. In the same review of case files, evaluators found that provid-ers
did not meet 42 percent of the families’ service needs. In almost
half of those cases, services were not provided because the family
failed to follow through with the referral.
Incomplete data limits DES’ ability to make informed judgments
about whether rates are appropriate or services are actually pro-vided.
When checked against the records in the DES database, 93
percent of the cases contained inaccurate information regarding the
type, number, or cost of services delivered. As a result, DES needs to
improve the quality and completeness of program information and
use it to determine the appropriateness of funding and service deliv-ery
rates.
Incomplete data limits
the ability to determine
whether rates are appro-priate
and all services
are delivered.
Summary
iv
OFFICE OF THE AUDITOR GENERAL
DES Should Improve Program
Oversight to Ensure All
Requirements Are Met
(See pages 29 through 33)
DES is not performing adequate oversight to ensure that providers
meet several requirements of the Pilot Program. In many cases, basic
demographic information about participants is inaccurate or un-known.
In 68 out of 80 cases reviewed, providers failed to collect or
accurately report demographic information about families being
served by the program. Omissions and errors occur, in part, because
until June 1999 providers were not required to update demo-graphic
information after the completion of assessments, when
much of the demographic information becomes available. Further,
until February 2000, DES was not aware that it was not receiving
updated information contained in the provider databases. In other
cases, providers have failed to collect or document required demo-graphic
information in either the client file or the program database.
In those cases, the lack of information appears to be an oversight on
the part of the worker completing the assessment. DES has ad-dressed
these issues through training and modification of required
forms. These efforts should be increased to ensure important pro-gram
data is collected.
Providers have also not obtained the Brief Family Assessment Scale
from most of the families in the program. Specifically, providers
obtained only 49 intake and exit assessment forms from 692 families
served by the program. As the assessments are useful in both case
planning and program evaluation, DES needs to better monitor
provider collection and use of the risk assessment and provide addi-tional
training as needed.
Finally, Local Advisory Boards, which are required under the stat-ute,
are not fulfilling all of their statutory responsibilities. The meet-ing
minutes from all Local Advisory Boards indicated that service
continuity has not been consistently addressed, that some boards
need to improve their oversight of program operations, and that
Board representation from both families and public agencies in the
community is difficult to attract and retain.
In many cases, basic infor-mation
is inaccurate or
unknown.
Summary
v
OFFICE OF THE AUDITOR GENERAL
Statutory Evaluation Components
(See pages 35 through 46)
Pursuant to Laws 1997, Ch. 223, §3, the Office of the Auditor General
is required to measure the program’s effect in reducing the number
of reports to Child Protective Services, the number of investigations
conducted by Child Protective Services, and the number of children
placed outside of the home. Although the number of reports CPS
received has declined since the Pilot Program's implementation, the
effect cannot be attributed to the Pilot Program since the report rate
has declined at the same or greater level in counties that did not
have the Family Builders Program.
The number of investigations performed by CPS has also declined.
There are at least two explanations for the decrease in CPS investi-gations.
First, all reports referred to the Family Builders Program no
longer require a CPS investigation. Second, the overall decline in
reports CPS received results in fewer investigations.
Further, the number of out-of-home placements statewide has in-creased
over the last three fiscal years with the major portion of the
increase occurring within the first two years. However, the percent-age
of Pilot Program families who later had a child removed from
the home did not differ from that of families who were referred to,
but did not receive, program services.
CPS reports have also de-clined
in counties without the
program.
vi
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
vii
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS
Page
Introduction and Background.............................. 1
Finding I: Family Builders Unable to
Show Impact on Child Safety
at This Time....................................................... 13
CPS Involvement Is Similar for Program
Participants and Comparison
Group Families..................................................................... 13
Most Pilot Program Participants
Begin with Low Risk for Abuse,
but Still Show Some Improvement..................................... 16
Recommendation................................................................. 17
Finding II: DES Needs to Examine
Costs and Number of
Services Delivered............................................ 19
DES Had to Set Capitated Rates
with Minimal Knowledge of What
Services Would Cost............................................................ 19
Provider Costs Fall Short
of DES Payment Rates ......................................................... 20
Many Services Identified
As Necessary Are
Not Being Provided ............................................................. 23
Inaccurate and Incomplete Service
and Billing Data Limits Ability to
Address Rate and Service Delivery Issues ......................... 24
Table of Contents
viii
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS (Cont’d)
Page
Finding II: (Cont’d)
Changes Are Needed
in Oversight and
Monitoring Efforts ............................................................... 25
Recommendations ............................................................... 27
Finding III: DES Should Improve
Program Oversight to Ensure
All Requirements Are Met................................ 29
Providers Do Not Obtain
and Accurately Report
Important Program Data ..................................................... 29
Local Advisory Boards
Do Not Meet All
Legislative Mandates ........................................................... 31
Recommendations ............................................................... 33
Statutory Evaluation Components....................... 35
Appendix ................................................................ a-i
Agency Response
Table of Contents
ix
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS (Cont’d)
Page
Figures
Figure 1 Family Builders Pilot Program Process .................. 5
Figure 2 Family Builders Pilot Program
Estimated Average Cost for Providing Services
October 1998 through June 1999............................. 21
Figure 3 Family Builders Pilot Program
Reported Instances of Child Abuse and Neglect
Years Ended June 30, 1996 through 1999 ............... 45
Tables
Table 1 Family Builders Pilot Program
Comparison of Percentage of Families
with Subsequent CPS Reports
Family Builders Participants vs. Comparison
Group Families
January 1, 1998 through June 30, 1999.................... ii
Table 2 Family Builders Pilot Program
Statement of Revenues, Expenditures,
and Changes in Fund Balance
Years Ended or Ending June 30, 1998,
1999, and 2000 (Unaudited)..................................... 8
Table 3 Family Builders Pilot Program
Providers, Service Areas, and Number and
Percentage of Families Served by Stage
January 1, 1998 through July 31, 1999..................... 9
Table 4 Family Builders Pilot Program
Comparison of Percentage of Families with
Subsequent CPS Reports
Family Builders Participants vs.
Comparison Group Families
Through June 30, 1999 ............................................. 15
Table of Contents
x
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS (Concl’d)
Tables (Concl’d)
Page
Table 5 Family Builders Pilot Program
Family Risk Scale at Entrance and Exit
Scores of Families with Closed
Service Plan Cases
January 1, 1998 through July 31, 1999..................... 17
Table 6 Family Builders Pilot Program
Providers, Counties, and
Maximum Contract Amounts
Years Ended or Ending June 30, 1998,
1999, and 2000........................................................... 39
Table 7 Family Builders Pilot Program
Family Risk Scale Assessment Areas
and Scores................................................................. a-ii
1
OFFICE OF THE AUDITOR GENERAL
INTRODUCTION AND BACKGROUND
The Office of the Auditor General has completed an evaluation of
the Family Builders Pilot Program. This evaluation was conducted
pursuant to the provisions of Laws 1997, Ch. 223, §3. This report
provides information regarding the Pilot Program’s implementation
and effectiveness in achieving its goals.
Child Maltreatment
Report Rate Increases
Beyond CPS Resources
During fiscal year 1997, the Arizona Department of Economic Secu-rity
(DES), Division of Child Protective Services (CPS) experienced a
34 percent increase in child abuse and neglect reports with no corre-spondent
increase in staffing. By the end of fiscal year 1997, 5,899
reports were uninvestigated. Though CPS investigated high- and
moderate-risk reports, low- and potential-risk reports received in
Maricopa and Pima Counties were often uninvestigated. At the
same time, concerns expressed by children’s advocates indicated the
need to provide more early intervention services to families. Ac-cording
to national statistics, 41 percent of the children who died
from child maltreatment are from families who had prior or current
contact with Child Protective Service agencies at the time of the
child’s death.1
Family Builders Emphasizes
Communities and Families in Solutions
To address the backlog of uninvestigated reports and prevent child
abuse and neglect, the Legislature and DES developed the Family
Builders Pilot Program. The program’s underlying principle is that
families coming to the attention of DES have different intervention
1 Prevent Child Abuse America. Child Abuse and Neglect Statistics. (April
1998). Web site: http://www.childabuse.org/facts97.html. [July 30, 1999].
In 1997, 5,899 CPS reports
went uninvestigated.
Introduction and Background
2
OFFICE OF THE AUDITOR GENERAL
needs and require flexible responses from DES and the community
in order to protect children and meet the family’s needs. While the
serious nature of moderate- to high-risk reports often requires a CPS
investigation that could lead to further intervention, low- and po-tential-
risk reports can be approached in a positive and supportive
manner, with needed treatment and other services provided imme-diately
without the trauma, stigma, and delay of the investigative
process. Child safety remains a priority, however. If the child’s safety
is in jeopardy or other information surfaces that would make an
investigation appropriate, Family Builders cases must be referred
back to CPS , the appropriate law enforcement agency, or both.
The Family Builders Pilot Program highlights the role of families
and communities by recognizing, accepting, and respecting the in-terdependence
and cultural diversity of the family and community
in problem-solving. The Family Builders philosophy includes the
following key components:
n Families are treated as partners with the provider agency in the
problem-solving process;
n Services focus on the family’s strengths rather than their prob-lems;
n Providers assist families in building support networks to reduce
reliance on formal support systems;
n Providers listen to all family members’ concerns, help them re-fine
those concerns, and assist families in establishing their own
goals for change;
n Services are accessible, available, and culturally sensitive; and
n Services are terminated when goals are achieved.
The approach is supported by research showing that child abuse
and neglect are related to the stresses of poverty or social disadvan-tages,
diminished social resources to manage those stresses, and
parenting practices. By identifying the family’s strengths and pro-viding
services that use those strengths to minimize stresses, the
Child safety remains a
priority.
Introduction and Background
3
OFFICE OF THE AUDITOR GENERAL
family gains empowerment and self-sufficiency. Therefore, it is be-lieved
that families will be less likely to have subsequent, more seri-ous
reports of abuse or neglect.
Because of the program’s new approach, DES received the Council
of State Governments’ 1999 Western Regional Innovations Award
for the Family Builders program.
Family Builders:
How It Operates
The Legislature established the Family Builders Pilot Program for
Maricopa and Pima Counties through Laws 1997, Ch. 223, §2. The
goal is to enhance parents’ ability to create safe, stable, and nurturing
home environments that promote safety of all family members and
healthy child development. The Pilot Program has been in existence
for two years. Unless the Legislature acts to continue the program, it
will be repealed on June 30, 2000.
The Pilot Program operates by responding to potential- and low-risk
reports of child abuse and neglect using a network of community-based
providers. The central CPS hotline receives all reports of child
abuse or neglect and prioritizes them into one of four categories,
ranging from potential to high risk. All moderate- and high-risk
reports, as well as any report that includes allegations of sexual
abuse, are automatically referred to a CPS investigator. Reports con-sidered
low or potential risk that are received in Maricopa or Pima
County go to a Family Builders coordinator at CPS, who decides
whether the report should be assigned for an investigation or re-ferred
to a Family Builders provider.
DES developed an eligibility and referral process to meet legislative
requirements. Foremost, the report must be classified as a low or
potential risk of abuse or neglect to be eligible for Family Builders. In
addition, a report classified low or potential risk with any of the
following characteristics is automatically disqualified from the Fam-ily
Builders program:
n Allegation of sexual abuse;
n Allegation of current injuries;
Reports must be classified
as a low or potential risk of
abuse or neglect to be eligi-ble.
Introduction and Background
4
OFFICE OF THE AUDITOR GENERAL
n Allegation involving a child that the court has found to be a
ward of the State due to the child’s incompetence or inability to
be restored to competency;
n Allegation concerning a child currently in an out-of-home place-ment;
n Allegation in which an immediate out-of-home placement is
required;
n Allegation regarding a parent under the age of 18 who is residing
with his or her parents as defined by the Family Assistance Ad-ministration;
n New report on a family that has an existing open case with CPS;
and
n Court-ordered investigation.
In addition, Family Builders coordinators review each report before
referring it to the Pilot Program to ensure that there are no aggravat-ing
factors that indicate a CPS investigation should be completed.
If the report is determined eligible for the Family Builders program,
the CPS Family Builders coordinator faxes the report to the provider
responsible for the area where the family lives. CPS closes all reports
that are referred to Family Builders and notes that the report was
referred to the program without a CPS investigation. A record of the
report is maintained for at least five years, as are unsubstantiated
reports investigated by CPS. Family Builders specialists do not con-duct
investigations, but they do respond to all referrals and assess the
safety of all children in the home. In addition, they are required to
refer cases back to CPS if there are signs of abuse or neglect.
Upon receipt of the referral, the Family Builders specialist, as a mem-ber
of the provider staff, begins the process of providing services to
the family as illustrated in Figure 1 (see page 5). The Family Builders
specialist informs the family that CPS received a report about them
and that CPS has referred the report to the Family Builders provider
to offer voluntary services to the family. If the family agrees to accept
services, the Family Builders specialist conducts an assessment to
identify the family’s strengths and any areas that the family would
CPS closes all reports that
are referred to Family
Builders.
Introduction and Background
5
OFFICE OF THE AUDITOR GENERAL
Figure 1
Family Builders Pilot Program Process
Family
Builders
Report
Closed
Report
Closed
If family declines
services, the report is
closed and is docu-mented
on the CPS
and Family Builders
databases.
If there are signs of
abuse or neglect, the
report is referred
back to CPS.
CPS forwards report to
Family Builders
specialist.
If family accepts, the
Family Builders Spe-cialist
performs a
Family Assessment.
Family Builders specialist
and family create a Family-
Centered Service Plan to
address the problems.
Option 1
Option 2
Option 3
If there are signs of
abuse or neglect, the
Family Builders spe-cialist
makes a report to
CPS.
Family Builders specialist at-tempts
2 unannounced visits
to the family within 48 hours
(excluding weekends) to offer
voluntary services.
Source: Auditor General staff analysis of Family Builders training material for policy and procedures.
Service Plan case is
closed when family no
longer desires services.
Introduction and Background
6
OFFICE OF THE AUDITOR GENERAL
like to improve. Clients can then choose to participate in developing
a service plan. This process allows the client to specify and accom-plish
small tasks; experience the success; and build the confidence
and skills needed to accomplish other tasks. The development of a
service plan is an ongoing process. The provider monitors the service
plan’s progress, conducts home visits to determine the safety of the
children in the home at the time of the visit, and determines the need
for continued services and contact with the family. Cases are closed
at any point when the family decides to end contact with the pro-gram.
The intent of the program is to close the case when the family
completes all the tasks and goals they wanted to accomplish as iden-tified
in the service plan.
The Family Builders provider staff provides services based on the
family’s needs. Family Builders providers are required to develop
local resources, which may be fee-for-service, contracts, or in-kind
collaborations to offer a wide range of services to meet families’
needs and are required to make available the following services, at a
minimum:
n Family assessments
n Case management
n Parenting skills training
n Parent aide services
n Housing search and relocation
n Emergency services
n Supportive intervention/guidance counseling
n Intensive family preservation services in the form of crisis coun-seling
and other emergency services
n Child daycare
n Transportation
n Respite services
Introduction and Background
7
OFFICE OF THE AUDITOR GENERAL
n Shelter services with parental consent
Families choose which services they want to receive; therefore, not all
families receive each of the above services. A typical family receives
between four to six months of case management, parent aide and/or
counseling, and some emergency services, such as food, clothing, and
rent or utilities assistance. However, some families received services
for over a year depending on their individual needs.
Finally, providers were required to establish Local Advisory Boards.
Both statute and provider contracts require that the Local Advisory
Boards ensure services are available when they are needed, provide
oversight to the program, and include representatives from families
in the community and local public agencies, such as DES, schools,
and the Arizona Health Care Cost Containment System.
Appropriations and Contracting
DES administers the Family Builders Pilot Program. The Legislature
appropriated over $12.4 million for fiscal years 1998 and 1999 for
Family Builders implementation. However, during fiscal years 1998
and 1999, the Legislature reallocated approximately $3.9 million of
the original appropriations to other DES programs due to the Pilot
Program’s lack of expenditures. Therefore, the final appropriation
for the Pilot Program was approximately $8.5 million for fiscal years
1998 and 1999, as illustrated in Table 2 (see page 8). In addition to
funding services for families, the appropriations specifically fund
one full-time contract specialist and two full-time Family Builders
Program specialists who are responsible for the Program’s adminis-trative
tasks, including training provider staff, observing provider
staff during home visits to clients, and conducting quarterly moni-toring
reviews of provider files. In addition, Family Builders pro-gram
specialists conduct regular quarterly meetings with providers
to discuss program updates and policy changes.
DES contracted with eight community-based social service agencies
in Maricopa and Pima Counties to provide services to the families.
Providers are paid a capitated rate (a set amount per service) for
each of three service delivery stages. The Family Builders Pilot Pro-gram
received 8,335 referrals between January 1, 1998, and July 31,
1999. Of those referrals, 5,578 (67 percent) families declined services
Twenty-eight percent of the
families completed service
plans to receive services.
Introduction and Background
8
OFFICE OF THE AUDITOR GENERAL
Table 2
Family Builders Pilot Program
Statement of Revenues, Expenditures, and Changes in Fund Balance
Years Ended or Ending June 30, 1998, 1999, and 2000
(Unaudited)
19981 1999 2000
(Actual) (Actual) (Estimated)
Revenues:
State General Fund appropriations2 $2,128,700 $6,378,600 $7,996,100
Expenditures:
Personal services and employee related 129,284 171,757 156,300
Travel, in-state 352 31,700
Aid to individuals3 1,898,110 5,703,502 7,767,100
Other operating 16,123 2,222 41,000
Total expenditures 2,043,517 5,877,833 7,996,100
Excess of revenues over expenditures 85,183 500,767
Fund balance, beginning of year      85,183 585,950
Fund balance, end of year4 $ 85,183 $ 585,950 $ 585,950
1 The Family Builders Pilot Program began in January 1998; therefore, the 1998 amounts are for only half
of the fiscal year.
2 Amounts shown are the final appropriations for the Family Builders Pilot Program. For 1998 and 1999,
the Legislature reduced the original appropriation by $2,277,900, and $1,628,700, respectively, and re-allocated
the monies to other Department of Economic Security programs.
3 Amounts shown are payments made to providers for services.
4 Fund balance may be reverted to the State General Fund if it is not used to pay claims that contracted
vendors failed to file in prior years. The Department of Economic Security expects most of the fund ba l-ance
to be used for such claims.
Source: Auditor General staff analysis of the Arizona Financial Information System Status of Appropria-tions
and Expenditures report, State of Arizona Appropriations Report, and financial information
provided by the Department of Economic Security for the years ended June 30, 1998 and 1999.
The Department estimated the fiscal year 2000 financial information.
Introduction and Background
9
OFFICE OF THE AUDITOR GENERAL
and 2,757 (33 percent) families received an assessment. Of those
referred, 2,326 (28 percent) families completed a service plan to re-ceive
services. DES contracts were based on the assumption that ap-proximately
75 percent of the families referred to providers would
agree to having an assessment performed and that 30 percent of the
families would have service plans completed. While most providers
met the expectation for service plan engagement, no provider met the
expectation for assessments. Table 3 presents the eight providers, their
service area, and the number and percentage of clients completing
each of the three stages of service delivery.
DES has recently expanded the Family Builders program to an ad-ditional
eight counties. The Pilot Program was implemented in
Yavapai, Coconino, Navajo, and Apache Counties in October 1999.
Graham, Greenlee, Cochise, and Santa Cruz counties implemented
the Pilot Program in November 1999.
Table 3
Family Builders Pilot Program
Providers, Service Areas, and Number and Percentage of Families Served by Stage
January 1, 1998 through July 31, 1999
Provider Service Area
Referrals
Completed
Assessments
Completed
Service Plans
Completed
Arizona Baptist
Children’s Services Northwestern Maricopa County 1,968 679 (35%) 623 (32%)
Arizona’s Children
Association Southwestern Maricopa County 1,063 297 (28) 229 (22)
Arizona Partnership
for Children
Southeastern Maricopa County
and Northeastern Pima County 2,505 747 (30) 606 (24)
Black Family and Child
Services South-central Maricopa County 563 179 (32) 157 (28)
CARENOW Northeastern Maricopa County 820 313 (38) 251 (31)
Child and Family
Resources Central Phoenix 319 112 (35) 106 (33)
Marana Unified School
District Northwestern Pima County 169 85 (50) 72 (43)
Our Town Family
Center Southeastern Pima County 928 345 (37) 282 (30)
Total 8,335 2,757 (33%) 2,326 (28%)
Source: Auditor General sta ff analysis of Family Builders provider contracts and the Family Builders program
database records for January 1, 1998 to July 31, 1999.
Introduction and Background
10
OFFICE OF THE AUDITOR GENERAL
Scope and Methodology
To evaluate the Family Builders Pilot program, evaluation staff en-gaged
in structured observations and conducted file reviews cover-ing
all providers. Office of the Auditor General evaluators also ana-lyzed
the measurements of risk and potential for child abuse at pro-gram
entrance and exit as identified by Family Builders casework-ers.
In addition, Family Builders participants’ involvement with CPS
was compared to the CPS involvement of other families who did not
participate in the program.1 Evaluators measured participants’ child
abuse report rates subsequent to program entry and compared these
rates with those of nonparticipants.
n Site Visits—Evaluation staff visited each of the eight Family
Builders providers. Site visits included accompanying staff on re-ferral,
assessment, service plan, and service delivery visits.
n Document Reviews—A review of 80 randomly selected open
and closed cases, 10 from each provider, was conducted to
document the accuracy of the database, to reconstruct costs, and
to determine the amount and type of services delivered. Cases
were selected from families who signed a service plan between
October 1, 1998 and March 31, 1999. Meeting minutes of the Lo-cal
Advisory Boards for each provider were reviewed from the
beginning of the program to July 1999.
n Assessment Tools—Two assessment tools, the Family Risk
Scale and the Brief Family Assessment Scale (BFAS), were used
to collect information about the risk for child abuse and neglect
from families who received services and whose cases were
closed before July 31, 1999. Family Builders caseworkers com-pleted
the Family Risk Scale to measure their perception of the
family’s risk for child abuse and neglect based on parent-centered,
child-centered, and economic-centered factors. Case-workers
began collecting the BFAS from families in October 1998
to measure the family’s perception of their strengths, resources,
and problem areas. The BFAS is a voluntary assessment and is
not requested from families who cannot read, speak, or write in
English. Both assessments should be completed immediately
1 The unit of analysis in these cases is the primary caregiver. Therefore, the
terms “Family Builders participants” and “Family Builders families” are used
interchangeably.
Introduction and Background
11
OFFICE OF THE AUDITOR GENERAL
following the intake assessment and again upon case closure.
The Appendix (see pages a-i through a-iii) includes full descrip-tions
of both tools.
n Comparison Groups—Evaluators collected and analyzed CPS
involvement data on the Family Builders participants and two
comparison groups who were eligible for, but not enrolled in, the
program. The Family Builders group consisted of 1,254 families
who accepted services between January 1, 1998 and December
31, 1998. The time period was selected to capture subsequent re-ports,
if any, for a minimum of six months following families’ in-volvement
with the Pilot Program.
4 The first comparison group included 2,995 families who were
referred to the Family Builders program in Maricopa and
Pima Counties between January 1, 1998, and December 30,
1998, but did not sign a service plan to accept services. The
families are similar to the participant group relative to the age
of the primary caregiver. The average ages of the Family
Builders and comparison group caregivers are 33.5 and 33.8,
respectively. The comparison group has a statistically signifi-cant
higher proportion of male caregivers (11.8 percent) than
the Family Builders group (9.3 percent). Also, there are sta-tistically
significant differences in the ethnicity of the families
in the comparison group and the Family Builders group.
Specifically, the comparison group has a higher proportion of
other/unknown ethnicity families, and fewer Hispanic fami-lies.
4 The second comparison group included 3,523 families served
by DES in locations other than those where the program had
been established. Because the Family Builders program was
piloted in Maricopa and Pima Counties, the second compari-son
group is comprised of families in counties other than
Maricopa and Pima. This group was selected from families
reported to CPS between January 1, 1998 and December 31,
1998. Evaluators determined them to be eligible for Family
Builders using the DES automatic disqualification criteria for
referring families to the Pilot Program. There are several sta-tistically
significant differences in the background and
demographic characteristics of the comparison group and the
Family Builders group. First, the comparison group has more
Introduction and Background
12
OFFICE OF THE AUDITOR GENERAL
male primary caregivers (20.8 percent) than the Family
Builders group (9.3 percent). Second, the primary caregivers
are slightly older; the average age of the primary caregiver is
35.5 in the comparison group, and 33.5 in the Family Builders
group. Finally, the comparison group has higher proportions
of White and American Indian families, and fewer Hispanic,
Black, and other/unknown ethnicity families.
Evaluators analyzed the number of prior and subsequent CPS re-ports
on the Family Builders participants and two comparison
groups. In addition, the number of out-of-home placements for the
Family Builders participants was compared with the proportion of
out-of-home placements for the families who declined services from
Family Builders, and the total number of out-of-home placements
reported by CPS.
n Findings—The research results were used to develop findings
and recommendations in the following four areas:
4 The need to continue and improve the collection of outcome
data so that program impact can be determined;
4 The need to collect accurate service and cost data and estab-lish
rates that correctly reflect the cost of providing services;
4 The need to better monitor the collection of required program
data; and
4 The need to improve the involvement of Local Advisory
Boards in program oversight and direction.
Acknowledgements
The Auditor General and staff express appreciation to the Director of
Economic Security, the staff of DES’ Division of Children and Family
Services, and the Family Builders program supervisors and staff for
their cooperation and assistance throughout the evaluation.
13
OFFICE OF THE AUDITOR GENERAL
FINDING I FAMILY BUILDERS UNABLE TO
SHOW IMPACT ON CHILD
SAFETY AT THIS TIME
The Family Builders Pilot Program does not show evidence of in-creased
child safety for participant families at this time. The pro-portion
of Pilot participants who received a CPS report following
their involvement with Family Builders is similar to the subsequent
CPS report rate of both comparison groups’ families. Although
analysis of assessments completed by caseworkers at the start and
end of program participation found that participant families exited
the program with a somewhat lower risk for committing child
abuse and neglect, no measurable impacts on child safety can be
attributed to the decreased risk. Attempts to further evaluate these
results through families’ own assessments of their stress levels
were unsuccessful because too few self-assessments had been
completed.
CPS Involvement Is Similar for
Program Participants and
Comparison Group Families
Participant and comparison group families had similar rates of
alleged and substantiated reports of child abuse and neglect fol-lowing
their initial report of child abuse or neglect, referred to
hereafter as the “referring report.” Specifically, there is no differ-ence
between the proportion of participant families who received
reports of child abuse or neglect subsequent to the “referring re-port”
and their involvement in Family Builders and the proportion
of comparison group families who later received a report. Moreo-ver,
there was no difference between the proportion of families in
the three groups who had subsequent substantiated CPS reports. In
addition, incomplete and inaccurate information about families
served by the program limits evaluators’ ability to explain the un-derlying
reasons for these results.
There is no difference in
subsequent CPS report
rates between program
families and the comparison
groups.
Finding I
14
OFFICE OF THE AUDITOR GENERAL
No meaningful difference in the proportion of participant and
comparison group families with subsequent reports—The review
of CPS report data from July 1, 1995, to June 30, 1999, found no
meaningful difference between the proportion of participant fami-lies
who received CPS reports after the receipt of the initial, refer-ring
report and the proportion of families from either comparison
group who subsequently received reports (see Table 4, page 15).
Specifically, 31.9 percent of participant families received one or more
CPS reports after being referred to the Family Builders program. In
comparison, 24.2 percent of families who were referred to the Pilot
Program, but did not receive services, received one or more subse-quent
CPS reports of child abuse or neglect. Similarly, 28.5 percent of
families served by DES in counties without the Family Builders Pilot
Program had subsequent reports of child maltreatment.
In addition, when looking only at families who had no CPS reports
prior to the referring report during this period, there is no meaning-ful
difference between the proportion of Pilot Program families and
the proportion of comparison families who had a subsequent report.
Participants have same rate of substantiated reports as comparison
group families—Moreover, there is no significant difference between
the proportion of participant families who have subsequent sub-stantiated
CPS reports and the proportion of families in either com-parison
group who later received a substantiated CPS report. As
shown in Table 4 (see page 15), 6.5 percent of Family Builders par-ticipants
have subsequent substantiated CPS reports compared with
5.7 percent of families who were referred to but did not receive pro-gram
services and 6.6 percent of families served by DES outside the
Family Builders service area.
When examining only those families with no reports prior to the
referring report, there was no significant difference in the proportion
of families who received substantiated reports subsequent to their
involvement with Family Builders or, as in the case of comparison
groups, to their initial, potentially referring report.
Limited ability to explain underlying reasons for these outcomes—
In attempting to examine the underlying reasons for these outcomes,
evaluators were hoping to use additional service and demographic
information about participants to further explain them. Specifically,
evaluators wanted to examine whether aspects such as length of
Finding I
15
OFFICE OF THE AUDITOR GENERAL
time in program, types of services received, family’s history of sub-stance
abuse and domestic violence, employment, income, health
care status, education, and other factors, could possibly be useful in
explaining why there was no meaningful difference in the CPS
involvement of pilot participants and the comparison group fami-lies.
Evaluators were unable to examine these factors at this time
Table 4
Family Builders Pilot Program
Comparison of Percentage of Families with Subsequent CPS Reports
Family Builders Participants vs. Comparison Group Families
Through June 30, 1999
Percentage of Families with
Subsequent Reports1
Group
All
CPS Reports3
Substantiated
CPS Reports Only
All families
Family Builders families 31.9% 6.5%
Families referred to, but not receiving,
Family Builders services4 24.2 5.7
Families outside Family Builders service area 28.5 6.6
Families without prior CPS Reports only2
Family Builders families 25.0 4.3
Families referred to, but not receiving,
Family Builders services4 17.9 3.5
Families outside Family Builders service area 23.6 5.1
1 Statistical tests were conducted to determine whether there was a meaningful difference in the proportion of Family
Builders participants compared to the proportion of comparison group families with subsequent CPS reports of child
abuse and neglect. Those tests are described in the Appendix, pages a-iii through a-v, and suggest that any observed di f-ferences
are due to chance.
2 Families with no CPS reports prior to the one that was referred to the Pilot Program or selected for the comparison
group.
3 Includes alleged and substantiated CPS reports.
4 These families were referred to the Family Builders program, but did not sign a service plan because they either refused
services, did not need services, could not be located, or were referred back to CPS. Some of these families did participate
in an assessment.
Source: Office of the Auditor General staff analysis of Child Protective Services data from July 1, 1995 through June 30,
1999.
Finding I
16
OFFICE OF THE AUDITOR GENERAL
because of the problems associated with the Family Builders data-base.
Much of the data was incomplete and, in many cases, inaccu-rate.
(See Finding III, pages 29 through 33.)
Most Pilot Program Participants
Begin with Low Risk for Abuse,
but Still Show Some Improvement
The Family Risk Scale results indicated that families in the Pilot Pro-gram
slightly reduced their risk for child abuse and neglect. Even
though Pilot Program families began with scores reflecting a low
risk for child abuse and neglect, risk assessments administered again
at the end of the program showed they made some improvement
after participating in the program. Unfortunately, too few families
completed the Brief Family Assessment Scale, which could have
allowed evaluators to better interpret the changes in family risk for
child maltreatment.
Family Risk Scale scores improved slightly—The Family Risk Scale
assessment results indicate that program families experienced a
slight but statistically significant decrease in risk in each of the three
measured areas. Using the Family Risk Scale, the caseworker as-sessed
the family's economic-centered, parent-centered, and child-centered
risk for child maltreatment at program intake and exit. The
assessment was conducted for 1,267 families. Numerous elements
make up each of the three scales. Respective examples of each type
of risk are financial problems, parent’s use of physical punishment,
and child’s mental health. Caseworkers rated the family’s risk on a
scale that ranges from one to four, five, or six, depending on the
scale. A one indicates adequate and low risk and higher numbers
represent increasing degrees of inadequacy and risk. As shown in
Table 5 (see page 17), families' risk declined in each of the three
measured areas.
While the average reduction in risk is small, program participants’
lower intake scores did not leave much room for improvement. On
average, caseworkers assessed the intake risk of program families to
be low on each of the individual dimensions that comprise the as-sessments’
three scales. For example, of 26 elements, there was only
one, financial problems, in which the participants received an aver-age
score above a 2 on a 6-point scale.
Scores show a slight but
statistically significant
reduction in the risk for
child abuse or neglect.
Finding I
17
OFFICE OF THE AUDITOR GENERAL
BFAS results are inconclusive—The BFAS results cannot be used to
assess the outcomes for the entire program because they do not ade-quately
represent those families who have been served by the pro-gram.
The BFAS measures a family’s perception of stress and the
resulting risk for child abuse and neglect in seven areas: personal
stress, family support, economic stress, aggressive behavior, child
problems, drug use, and alcohol use. Providers obtained pre- and
post-assessments from only 49 of the 692 families who participated
in the program (approximately 7 percent). Finding III (see pages 29
through 33) discusses the reasons for the low number of BFAS as-sessments
and emphasizes the need to ensure a higher completion
rate of the assessments. For the few families who did complete the
assessments, the differences between intake and exit scores were
statistically significant for only two of the seven BFAS subscales
(personal stress and economic stress). On both of these scales, the
risk for child abuse and neglect decreased.
Recommendation
1. DES should ensure that accurate and complete program infor-mation,
such as service delivery data, demographic data, and
family risk assessments, is collected as specified in provider con-tracts,
training, and existing forms, to ensure that program out-comes
can be fully measured in the future.
Table 5
Family Builders Pilot Program
Family Risk Scale at Entrance and Exit
Scores of Families with Closed Service Plan Cases
January 1, 1998 through July 31, 1999
Average Risk Score
Risk Assessment Area Entrance Exit Change1
Economic risk factors 1.61 1.40 -0.21
Parent-centered areas 1.71 1.60 -0.11
Child-centered areas 1.50 1.35 -0.15
1 For all three risk areas, the reduction in risk is statistically significant at the p<.05 level. That
is, the probability that the change in risk scores occurred by chance is less than 5 in 100.
Source: Auditor General staff analysis of pre and post scores on Family Risk Scales provided
by Family Builders caseworkers.
18
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
19
OFFICE OF THE AUDITOR GENERAL
FINDING II DES NEEDS TO EXAMINE
COSTS AND NUMBER OF
SERVICES DELIVERED
DES currently lacks the necessary information to determine if pro-viders
are delivering adequate services and are being paid at appro-priate
rates. Based on a review of 50 families who signed a service
plan, there are indications that the rates being paid to many provid-ers
may be higher than the costs of the services provided, and there
are also indications that many clients may not be receiving the level
of services they need. However, problems with the quality and
completeness of service and billing data limit the ability to make
informed judgments about whether rates and service levels are ap-propriate.
DES needs to improve the quality and completeness of
this information and use it to determine if program funding or serv-ice
levels should be changed.
DES Had to Set Capitated Rates
with Minimal Knowledge of What
Services Would Cost
Because the Pilot Program was new and the costs unknown, DES set
the maximum rate for services based on an evaluation of the cost to
serve families through another DES program, the Family Preserva-tion
Program. The Family Preservation Program delivers services
similar to those required by the Family Builders program and has a
per-child cost of $1,483, according to DES estimates. Based on an
estimate of 2.4 children per household and additional referral and
assessment costs, the Pilot Program rate was set at a maximum of
$3,850 per family. This rate is divided into three components, one for
each stage of service delivery: referral, assessment, and service plan.
Providers are paid the capitation rate for completing each of the
three stages per client, regardless of the amount it costs them to per-form
those services.
Capitated rates negotiated with the eight providers ranged from
$2,030 to the maximum of $3,850. As a part of the bid process, pro-viders
proposed the amount it would cost to complete each of the
The maximum rate for
services was set at $3,850
per family.
Finding II
20
OFFICE OF THE AUDITOR GENERAL
three stages of service delivery for families referred to them. DES
published the maximum reimbursement rates for each service stage,
and four of the eight providers submitted bids that were at the
maximum. DES made the awards based on price and the quality
and scope of the proposed service delivery.
Provider Costs Fall Short
of DES Payment Rates
Reviews of 80 randomly selected cases found that the average
documented cost to provide services is less than the amount DES
paid for those services. Even when costs were close to the rates DES
paid, the reason was often related to the internal cost the provider
charged for the service, not to the number of services provided.
Average cost of services is less than capitated rate—Evaluation
staffs’ analysis showed that most providers’ costs for serving clients
were far below the capitated payments they received. Evaluation
staff reviewed case notes, bills, receipts, and service authorizations
from 80 cases randomly selected cases (ten from each of the eight
providers) to determine the cost of services provided to those fami-lies.
Overall, the average cost for providing services in the 80 cases
reviewed was $1,321 for closed cases and $1,675 for open cases,
which is between $1,500 and $1,850 less than the average capitated
rate paid for those services.1 The provider-by-provider average costs
are shown in Figure 2 (see page 21).
The costs in Figure 2 (see page 21) were adjusted to include esti-mated
referral and assessment costs, and a 10 percent factor for
overhead. Evaluators also estimated the average costs for families
who do not sign service plans to exceed provider payment rates by
no more than $90, on average.
1 The estimated range for closed cases is $1,089 to $1,553 and for open cases is
$1,343 to $2,008 (at a confidence level of 95 percent).
The average documented
cost for providing services
to the 80 cases sampled
was between $1,500 and
$1,850 less than the aver-age
capitated rate.
1,246
2,065
2,395
1,040
850
2,030
1,2011,133
3,850
722
1,071
3,850
1,064
1,346
3,850
2,5072,437
3,340
1,455
1,107
2,300
2,629
1,794
3,850
-
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
Figure 2
Family Builders Pilot Program
Estimated Average Cost for Providing Services1
October 1998 through June 1999
Finding II
OFFICE OF THE AUDITOR GENERAL
Arizona Baptist Arizona’s Arizona Black Family CARENOW Child and Marana Our Town
Children’s Services Children Partnership and Family Unified School Family
Association for Children Child Services Resources District Center
Dollars
Estimated Cost Amount Paid by DES
Open
Cases2
Closed
Cases3
1 Open and closed cases were included in the review in order to account for the differences that result from open cases that may include longer engagement with clients.
In addition, the cost associated with open cases may increase as services continue to be provided in the cases.
2 Open cases were those receiving services at the time of the file review.
3 Closed cases were those no longer receiving services from the Family Builders program at the time of the review.
Source: Auditor General staff analysis of provider contracts and program costs documented in case files for 10 randomly selected cases from each of the 8 providers.
21
Finding II
22
OFFICE OF THE AUDITOR GENERAL
Differences in average costs do not necessarily represent differences
in service levels—As Figure 2 (see page 21), shows, providers varied
substantially in the degree to which their costs were close to or dif-ferent
from their capitated rates. Analysis of the documentation
behind these costs showed that providers who had a much smaller
gap between average costs and their capitation rates did not neces-sarily
provide more services. In some cases, they simply listed much
higher per-unit costs for the services they provided. Three of the
eight providers stood out in that regard:
4 The average cost for Arizona Baptist Children’s Services was
strongly affected by one case in which the family counseling ex-pense
was $3,440. In contrast, the highest average cost for coun-seling
across all eight providers was $284, and the next highest
single case among the 80 cases analyzed was $858. The billing for
this case reflected a much higher per-session amount for coun-seling
than any of the other cases examined.
4 Our Town Family Center indicated that its case management
costs averaged $865 per case for case management and $450-600
per case for assessments. By contrast, average costs reported by
the seven other providers ranged from $335 to $706 for case
management and from $272 to $352 for assessments.
4 Child and Family Resources, during the months included in
the review, charged a case management rate between $138 and
$198 per hour, which is between $75 and $135 more per hour
than the next highest case management rate billed by any other
provider.
In most other respects, variations in average cost resulted largely
from differences in amounts and types of services provided and
from relatively small differences in associated rates. For example,
both Arizona’s Children Association and Black Family and Child
Services have lower case management rates than the other providers
and had few requests for emergency services in the costly forms of
rent assistance and utilities payments. Therefore, their average cost
to provide services is lower than that of other providers.
Some providers had much
higher per-unit costs for
services than other provid-ers.
Finding II
23
OFFICE OF THE AUDITOR GENERAL
Many Services Identified
As Necessary Are
Not Being Provided
One consideration in examining the differences between the capi-tated
rates paid to providers and the costs incurred in providing
services is whether the costs are low because needed services are not
delivered. Analysis of the 80 cases shows services identified as nec-essary
are often not provided. In all, 42 percent of the service needs
identified during the assessment process were not provided. The
reasons, to the extent they could be determined, are various and are
often outside the providers’ control.
Forty-two percent of service needs not met—The 80 cases analyzed
involved a total of 448 identified service needs, ranging from coun-seling
and parenting skills training to relocation and shelter services.
In all, 188 of the 448 identified service needs were not provided. In
other words, 61 of the 80 families did not receive all of the requested
services. They included services such as housing search and reloca-tion,
parenting skills training, counseling, transportation, and job
search services.
Reasons include inaction from both the family and the provider—
Families did not receive the requested services for a variety of
reasons. The family failed to follow through in approximately 48
percent of the cases in which services were not provided.
Other reasons related more to problems in service delivery. For ex-ample,
7 percent of needed services were not provided due to un-availability
and 11 percent were not provided because the provider
either failed to follow through or made an inappropriate referral. An
example of the latter reason would be referring the family to a serv-ice
provider that the family had expressed dissatisfaction with in the
past.
Additional reasons, such as the family no longer needing a service or
the case remaining open, account for 13 percent of unprovided
services. As discussed later, providers did not adequately document
the reasons why 21 percent of requested services were not provided.
Some of the requested serv-ices
not received included
parenting skills training,
counseling, and transporta-tion.
Finding II
24
OFFICE OF THE AUDITOR GENERAL
Inaccurate and Incomplete Service
and Billing Data Limits Ability to
Address Rate and Service Delivery Issues
While the review of costs and services indicates cause for concern,
problems with the quality and completeness of service and billing
data limit the ability to make informed judgments about whether
rates and service levels are appropriate. DES does not have accurate
information on the type, amount, and cost of services provided to
families. DES also lacks information to assess the appropriateness of
the level of services being provided to program participants.
Provider billing does not accurately reflect services provided—Pro-viders
are not supplying DES with complete and accurate informa-tion
about service delivery and costs. DES requires providers to
indicate the type and cost of services families receive each month in
the billing portion of the database. However, in 74 out of 80 cases
reviewed, providers failed to accurately report either the type, num-ber,
or cost of services delivered.
The differences in the file and the database appeared to be random,
with billing records for some months containing no information
about services delivered and the next month’s billing containing
different service amounts or costs other than those recorded in the
file. These errors are significant because incomplete and inaccurate
service and cost information limits the program’s ability to evaluate
cost based on all participants served, which is a critical management
tool for determining how to offer services and establishing a reason-able
rate to pay for them.
Data insufficient to determine appropriateness of service level—
Inaccurate and incomplete service data limits the ability to
determine if additional services should be provided to families. As
Finding I (see pages 13 through 17), pointed out, the program has
not yet shown an impact at the current level of services. However,
because accurate data is limited, the amount and type of services
could not be clearly linked to better outcomes. In the 80 cases
reviewed, providers failed to document the reasons needed services
were not provided 21 percent of the time. In one case the
caseworker’s notes indicated that the family expressed an interest in
counseling and childcare, but neither service was ever provided and
the case notes did not explain why.
In 74 of 80 cases, providers
failed to accurately report
either the type, number, or
cost of services.
Finding II
25
OFFICE OF THE AUDITOR GENERAL
Lack of complete and accurate data affects overall ability to evalu-ate
rates and service delivery—Missing and incomplete service need
and delivery information impacts case management and program
evaluation. Adequate documentation is critical to managing the
needs of families served by the program. Failure to document types
of services delivered or to identify why needed services were not
provided hinders the program’s ability to effectively address client
needs and access to services, especially when caseworkers change
during the course of service delivery. In addition, not knowing why
identified service needs are not delivered makes it impossible to
evaluate unmet service needs. For example, without documentation,
the Pilot Program cannot determine if services, such as the child care
or counseling identified by the client in the example above, are not
being delivered because families fail to follow through, because the
services are not available, or because workers need more training.
With adequate documentation, all of those issues could be ad-dressed
to better ensure that services are provided as needed.
Changes Are Needed
in Oversight
and Monitoring Efforts
Several aspects of DES’ monitoring and oversight of providers’ per-formance
need improvement. DES has only recently begun to com-pare
cost reports to provider files and the effectiveness of future
reviews will rest on improving the completeness and quality of in-formation
submitted. To ensure that this information is complete
and accurate, DES should continue to resolve confusion concerning
how costs are to be reported. Finally, DES needs to ensure that this
information is used to ensure that needed services are provided and
to set reimbursement rates that reflect the cost of providing these
services.
Reviewing cost information—Although DES has required providers
to submit complete and accurate billing information on a monthly
basis, it has only recently performed detailed reviews of those re-ported
costs. In September 1999, DES began reviewing the costs
associated with 90 open and closed case files representing all of the
providers. This review was ongoing at the time of this report and the
results could not be included here. Continuing reviews of this in-formation
are important as part of the quarterly monitoring of pro-
DES has recently begun to
compare cost reports to pro-vider
files.
Finding II
26
OFFICE OF THE AUDITOR GENERAL
vider documentation and operations. However, it will be of limited
usefulness unless the data quality is improved. As part of this proc-ess,
it will be important for DES to ensure that the billing and pro-gram
database allows providers to record all of their costs associated
with performing services for families, including referral and over-head
costs.
Improving guidance to providers—Providers have said that the
forms and database for recording costs and services are confusing. In
response, DES is currently revising the forms to correspond more
closely to the database. DES is also in the process of forming a task
force to study cost issues, including how to accurately reflect costs in
the database. Given the lack of completeness and accuracy in the
information currently being received, creating a manual that would
help providers submit such information appears to be an important
step.
Monitoring service delivery and assessing reimbursement rates—
Given the importance of service delivery and cost documentation,
DES needs to provide greater oversight of provider reporting and
tracking of service and cost data. This review needs to include
determining the extent to which needed services are delivered, and
documenting the reasons when they are not. In reviewing the
reasons why needs were not met, DES can determine if additional
service providers are needed or if workers need more training to
find services and motivate clients to follow through with services.
Finally, DES should determine an appropriate reimbursement rate
for services based on providers’ actual cost to provide all needed
services.
DES needs to determine the
extent to which needed serv-ices
are delivered.
Finding II
27
OFFICE OF THE AUDITOR GENERAL
Recommendations
To ensure appropriate reimbursement rates and service levels, DES
should:
1. Modify its billing forms and database to ensure that all provider
costs are captured; specifically, referral and overhead costs.
2. Develop a manual for reporting costs to ensure providers accu-rately
collect and report costs.
3. Include the following procedures in its quarterly monitoring
process:
a) Verify that the type, number, and costs of services reported in
the database reflect what is documented in the file; and
b) Ensure that providers’ files adequately explain the reasons
identified service needs are not met.
4. Examine the reasons that identified service needs are not being
provided and address those concerns through developing serv-ice
availability or training.
5. Use the improved service and cost information to regularly re-view
the appropriateness of capitated reimbursement rates so
that new rates can be set during the fiscal year 2001 contract ne-gotiation,
if appropriate.
28
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
29
OFFICE OF THE AUDITOR GENERAL
FINDING III DES SHOULD IMPROVE PROGRAM
OVERSIGHT TO ENSURE ALL
REQUIREMENTS ARE MET
DES is not performing adequate oversight to ensure that providers
meet several requirements for the Pilot Program. Basic demographic
information about participants, along with assessments of the level
of risk present in their family situations, is incomplete. Local Advi-sory
Boards, which are required under the statute, are not fulfilling
all of their statutory responsibilities. DES needs to better enforce
these requirements to help ensure that the Pilot Program is operat-ing
appropriately.
Providers Do Not Obtain
and Accurately Report
Important Program Data
Family Builders providers are not obtaining and reporting some
required information from program participants. As a result, the
Family Builders database contains inaccurate and incomplete infor-mation
about the families served by the program. Providers are also
not consistently obtaining assessments of family stress and resulting
risk to children, information that is important both for managing
cases and for evaluating what the Program is accomplishing.
Required demographics information is missing or inaccurate—
Demographic information is often not obtained or updated in the
database. Evaluation staff reviewed a total of 80 cases and found that
in 68 cases, the database and/or the files contained incorrect or incom-plete
demographic information. The omissions and errors occur for
three main reasons.
4 Partial or incomplete information is often obtained during
intake. DES requires providers to submit demographics infor-mation
about participants in order to receive payment for the re-ferral
portion of the capitated fee. For families who decline
The Family Builders data-base
contains inaccurate
and incomplete information.
Finding III
30
OFFICE OF THE AUDITOR GENERAL
assistance and therefore do not enter the subsequent assessment
phase, the information obtained during referral is the only infor-mation
available. This information is often incomplete. For those
who do receive an assessment, the information is not always up-dated
in the database. Further, until June 1999, providers were
not required to update demographic information collected at the
time of assessment.
Until February 2000, DES was not aware that it was not receiving
updated demographic information providers maintained in their
copy of the database.
4 Providers do not consistently obtain all demographics in-formation
from all families who continue into the assess-ment
phase. As a result, some information, such as marital
status, employment, or health insurance status, remains un-known.
Inaccurate demographics information could limit the
program’s ability to fully serve clients. For instance, if the pro-vider
does not determine that the family has no health insurance,
they cannot help the family address that issue before any
healthcare needs arise.
Although DES has taken steps to improve data collection, more
needs to be done. In addition to quarterly monitoring of each pro-vider,
DES has made the following improvements: (1) adding more
training about completing required forms and entering data into the
database and (2) requiring providers to update demographic infor-mation
on families who complete assessments. To further aid pro-viders,
DES needs to develop a user’s manual explaining the proce-dure
for recording information in the database. DES also needs to
include a review of the demographic information in its quarterly
monitoring.
Brief Family Assessment Scales not obtained from many partici-pants—
Although providers completed the Family Risk Scale for
most families, they have not obtained the Brief Family Assessment
Scale from most families who chose to participate in the program.
DES and Auditor General staff identified the Brief Family Assess-ment
Scale (BFAS) as a case management tool for caseworkers and a
means to measure the change in resources, strengths, and stressors
Finding III
31
OFFICE OF THE AUDITOR GENERAL
for families participating in the program. However, providers ob-tained
only 49 intake and exit BFAS scores from 692 families served
by the program after the BFAS began to be administered in October
1998.
A number of factors may have contributed to the low number of
participants with risk assessments, but providers did not consistently
record reasons why the assessment was not completed. Based on
interviews with provider staff, the factors include:
4 Losing contact with the family and therefore not being able to
obtain the closing assessment;
4 Refusal by the family to complete the assessment;
4 Caseworkers’ concern that asking families to complete the BFAS
would offend them, and;
4 Participants who do not speak or read English do not complete
the BFAS because there is not a Spanish version of the assessment.
To ensure that important risk assessment information is collected for
both case management and evaluation purposes, DES needs to con-tinue
monitoring how providers are using it. The BFAS is useful for
reporting changes in parenting stress levels and the consequent risk of
child abuse. DES needs to continue training providers on the impor-tance
of the BFAS and emphasize the importance of encouraging
families to complete it.
Local Advisory Boards
Do Not Meet All Legislative Mandates
Local Advisory Boards (Boards), which the law requires each pro-vider
to establish, have not fulfilled all of their responsibilities as speci-fied
in legislation and contract. These responsibilities include:
4 Ensuring that services are available when needed;
4 Providing direction to the program; and
4 Representing the community in which program families live.
Finding III
32
OFFICE OF THE AUDITOR GENERAL
Evaluation staff found deficiencies in all three areas.
4 Boards not ensuring service continuity—Boards do not fully
ensure that all services are available when needed. Although most
Boards have provided information on services available in the
community, there is little indication that they ensure those services
address the gaps in services experienced by providers or that pro-viders
consistently inform Boards of services they are having diffi-culty
providing.
4 Some Boards need to improve oversight—Most Boards have
participated in program oversight through such activities as re-viewing
quarterly reports, requesting analysis of program par-ticipation,
creating brochures, and making policy decisions.
However, one Board did not appear to meet regularly enough to
make any impact on the program and another Board never pro-vided
any direction other than to recommend additional training
for staff.
4 Membership does not meet requirements—Most Boards have
not been able to recruit and involve the required membership.
For instance, half of the providers have been unable to recruit
Board members from families in the community. Although all
Boards currently include a representative from local public agen-cies,
these members have also been difficult to attract and keep
involved for most Boards. In addition, those Boards that do have
the required membership have not been able to get members to
regularly attend meetings.
To improve Board activity and involvement, DES needs to provide
more-thorough review of Board membership and activities and
provide assistance with recruiting and involving Board members as
needed.
Many Boards have not been
able to recruit Board members
from families in the commu-nity.
Finding III
33
OFFICE OF THE AUDITOR GENERAL
Recommendations
1. To ensure that accurate and complete demographics information
is collected on program families, DES should:
a) Develop a user’s manual that details data collection proce-dures;
and
b) Compare the information reported in the database with in-formation
collected from file reviews during the quarterly re-view
process.
2. To improve usage of the BFAS, DES should:
a) Give additional training to providers on using the BFAS as a
case management tool and on encouraging families to com-plete
the assessment; and
b) Monitor the collection and use of the BFAS as part of the
quarterly review process.
3. To ensure that Local Advisory Boards fulfill their statutory and
contractual requirements, DES should conduct reviews of Board
activities and provide assistance with recruiting and involving
Board members.
34
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
35
OFFICE OF THE AUDITOR GENERAL
STATUTORY EVALUATION COMPONENTS
Pursuant to Laws 1997, Ch. 223, §3, the Office of the Auditor General
is required to include the following information in the Family Build-ers
program evaluation.
B.1. Information on the number of reports referred to the pilot
programs.
Between January 1, 1998 and July 31, 1999, 8,335 reports con-cerning
18,283 children were referred to the Pilot Program.
Approximately 75 percent of referred families reside in Mari-copa
County and the remaining 25 percent reside in Pima
County.
B.2. Information on the number of families served.
Information on the number and characteristics of program
participants was available for families through July 31, 1999.
Since the January 1, 1998, implementation of Family Builders,
2,757 families received assessments, and 2,326 families, repre-senting
5,723 children, agreed to accept services. Approxi-mately
75 percent of families served reside in Maricopa
County and the remaining 25 percent reside in Pima County.
B.3. Demographic information on the families served.
Demographics are reported on all 2,326 families served with
limitations as described in Finding III (see pages 29 through
33).
4 Caregivers—The majority of primary caregivers in the
program are female, with fewer than 10 percent being
male. In 97 percent of the cases the caregivers are parents.
Other relatives care for almost all of the remaining 3 per-cent
of program children. The average age of female care-givers
is 33. The average age of the male caregivers is 35.
Statutory Evaluation Components
36
OFFICE OF THE AUDITOR GENERAL
4 Caregivers’ Marital Status and Living Arrangements—
While 62 percent of the primary caregivers reported being
single or separated at the time of enrollment, less than
half reported living alone with their children. In addition
to the 24 percent who reported living with a spouse, 14
percent of caregivers live with their parents or another
relative and 17 percent live with a non-relative or co-habitating
partner.
4 Caregivers’ Education and Employment—Approxi-mately
75 percent of the caregivers have either a high
school diploma or GED. In addition, approximately 20
percent of the caregivers report having post-secondary
education or training. However, only 48 percent of female
and 71 percent of male caregivers reported being em-ployed
either full-time or part-time.
4 Caregivers’ Ethnicity—Ethnicity among primary care-givers
varies. Approximately 60 percent of the caregivers
are Anglo, 27 percent are Hispanic, 7 percent are African-
American, and 3 percent are Native American. “Other”
makes up the balance.
4 Income—The majority of program participants belonged
to impoverished households. The median income is
$16,200 per family among family households whose an-nual
household income, including assistance, was re-ported.
Approximately 30 percent were below $12,000; 30
percent were between $12,000 and $20,000; and 40 per-cent
were above $20,000. While most families reporting
their source of income indicated wages as a source, ap-proximately
42 percent of families depended on one or
more welfare benefits, such as food stamps, social secu-rity,
and TANF to either support their family or to sup-plement
their income.
4 Family size—One-quarter of the families in the program
have only one child. Thirty-one percent of the families
have two children, 23 percent have 3 children, and the
remaining 22 percent have four or more children.1
1 Total of percentages equals more than 100 percent due to rounding.
Statutory Evaluation Components
37
OFFICE OF THE AUDITOR GENERAL
4 History of substance abuse and domestic violence—
Some of the families in the Pilot Program also have sub-stance
abuse and domestic violence problems that can
make the challenges of parenting more difficult. Specifi-cally,
16 percent of females and 26 percent of males re-ported
abusing alcohol or drugs within 6 months of pro-gram
intake. In addition, 29 percent of families reported
the occurrence of domestic violence in the home within 6
months of program intake.
B.4. Information on the type of services provided and the
families’ satisfaction with the services.
n Services Delivered—All providers are required to pro-vide
assessment, case management, child care, housing
search and relocation, parenting skills training, suppor-tive
intervention and guidance counseling, transporta-tion,
emergency, intensive family preservation, parent
aide, respite, and shelter services. Because the database
recording the types of services delivered is not accurate
(see Finding II, pages 19 through 27) the specific type of
services cannot be determined for all cases. Based on the
evaluators’ review of 80 randomly selected cases, all
families received case management, 29 percent received
counseling services, 31 percent received parenting skills
training or parent aide services, and 56 percent received
emergency services such as food, clothing, or utilities,
rent, and phone payments. In addition, some families re-ceived
transportation, housing search and relocation,
child-care, and shelter services.
n Client Satisfaction—Because only 7 percent of served
families responded to the client satisfaction survey, client
satisfaction with the program could not be fully deter-mined.
Family Builders specialists provided families with
a confidential survey to measure their satisfaction with
the services they received, the program, and the workers.
Families were also given a postage-free return envelope
to mail the completed survey to the Office of the Auditor
General. However, during the first two years of the pro-gram,
the satisfaction survey was not available in Span-ish.
Out of 2,326 families who were served by the pro-
Statutory Evaluation Components
38
OFFICE OF THE AUDITOR GENERAL
gram, only 170 returned a satisfaction survey. In addition,
providers are not equally represented in the results.
Specifically, four providers had three or fewer surveys re-turned
by their clients.
The 170 participants who did respond to the survey were
satisfied with the program. For example, 91 percent of the re-spondents
felt their family was somewhat or a great deal
better off as a result of their involvement with Family Build-ers.
In addition, 92 percent reported feeling that they were a
great deal or somewhat better able to deal with their prob-lems
as a result of the Family Builders program. Finally,
fewer than 2 percent of families reported that Family Builders
was unable to provide services to meet their family’s needs.
B.5. Information on the cost of services provided.
As discussed in Finding II (see pages 19 through 27), the cost
of services cannot be fully determined. DES paid approxi-mately
$7,601,612 to providers to accept 8,335 referrals, per-form
2,757 assessments, and provide extended services to
2,326 families. Providers receive capitated payment for com-pleting
each service stage regardless of the amount it costs
them to perform those services. Capitated rates vary by pro-viders
as some providers bid different rates. In addition, each
provider incurs different costs for the same types of services
since the internal rates depend on which agency is providing
the service. However, the providers failed to accurately rec-ord
costs in their billing statements and the database; there-fore,
actual cost of services cannot be determined for the en-tire
population.
As reported in Finding II, the evaluators’ review of 80 ran-domly
selected cases indicates that the actual cost of services
for closed cases ranged from $850 to $2,437, and for open
cases ranged from $722 to $2,629. Both ranges include an es-timate
for under-documented referral, assessment, and over-head
costs.
Statutory Evaluation Components
39
OFFICE OF THE AUDITOR GENERAL
B.6. Information on providers.
DES awarded contracts to eight providers in Maricopa and
Pima Counties, as illustrated in Table 6 (see page 39). Provid-ers
formed fee-for-service, contract, and in-kind collabora-tions
with other social and health service providers in their
local communities to provide services to families. DES’ con-tract
with providers included an expectation that providers
would perform assessments for approximately 75 percent of
all families referred to them and provide service plans to ap-proximately
30 percent of all families referred to them. While
most providers met the expectation for service plans, no pro-vider
completed the expected percentage of assessments.
n Arizona Baptist Children’s Services—Arizona Baptist
Children’s Services serves the northwestern Maricopa
County area. They have 56 collaborators, including Par-ents
Anonymous, Maricopa Skill Center, Crisis Nursery,
Salvation Army, The Bridge, Treatment Assessment
Table 6
Family Builders Pilot Program
Providers, Counties, and Maximum Contract Amounts1
Years Ended or Ending June 30, 1998, 1999, and 2000
Provider County 19982 1999 2000
Arizona Baptist Children’s Services Maricopa $ 582,205 $1,164,410 $1,164,410
Arizona’s Children Association Maricopa 289,610 579,220 579,220
Arizona Partnership for Children Maricopa and Pima 1,038,350 2,076,700 2,076,700
Black Family and Child Services Maricopa 267,350 534,700 534,700
CARENOW Maricopa 375,550 606,100 606,100
Child and Family R esources Maricopa 134,310 268,620 268,620
Marana Unified School District Pima 65,100 130,200 130,200
Our Town Family Center Pima 535,500 1,071,000 1,071,000
Total $3,287,975 $6,430,950 $6,430,950
1 Provider contracts establish a rate for each unit of service (referral, assessment, and service plan) not to exceed the maxi-mum
number of units indicated for each contract service.
2 The Family Builders Pilot Program began in January 1998; therefore, the 1998 amounts are for only half the fiscal year.
Source: Auditor General staff analysis of Family Builders provider contracts for fiscal years 1998, 1999,
and 2000.
Statutory Evaluation Components
40
OFFICE OF THE AUDITOR GENERAL
Screening Center, and several local churches. In addition,
Arizona Baptist Children’s Services employs 9 casework-ers
to perform assessments and provide services to their
clients. Arizona Baptist Children’s Services performed as-sessments
on 35 percent of all families referred to them
and provided service plans to 32 percent of all families re-ferred
to them.
n Arizona's Children Association—Arizona's Children
Association serves the southwestern Maricopa County
area. Their four collaborators are Florence Crittenton,
Southwest Behavioral Health Services, Southwest Hu-man
Development, and Tumbleweed. Five caseworkers
conduct assessments and provide services to their clients.
Arizona's Children Association performed assessments
on 28 percent of all families referred to them and pro-vided
service plans to 22 percent of all families referred to
them.
n Arizona Partnership for Children—Arizona Partnership
for Children serves families in southeastern Maricopa
County and northeastern Pima County. They have 23
collaborators, including Parents Anonymous, Devereux-
LaHacienda, The Blake Foundation, and Touchstone
Community. Nineteen caseworkers conduct assessments
and provide services to their clients. Arizona Partnership
for Children performed assessments on 30 percent of all
families referred to them and provided service plans to 24
percent of all families referred to them.
n Black Family and Child Services, Inc.—Black Family
and Child Services serves families in south-central Mari-copa
County. They have 16 collaborators, including
Goodwill, YMCA, Salvation Army, Sojourner Center,
Phoenix Day Child Development Center, and Maricopa
Skills Center. Five caseworkers conduct assessments and
provide services to their clients. Black Family and Child
Services performed assessments on 32 percent of all fami-lies
referred to them and provided service plans to 28
percent of all families referred to them.
Statutory Evaluation Components
41
OFFICE OF THE AUDITOR GENERAL
n Carenow—Carenow serves families in northeastern
Maricopa County. They have 5 collaborators: Jewish
Family and Child Services, PREHAB of Arizona, Valle del
Sol, Youth Evaluation and Treatment Centers, and West
Side Social Services. Nineteen caseworkers conduct as-sessments
and provide services to their clients. Carenow
performed assessments on 38 percent of all families re-ferred
to them and provided service plans to 31 percent of
all families referred to them.
n Child and Family Resources—Child and Family Re-sources
serves families in Central Phoenix. They have
eight collaborators, including City of Phoenix/Young
Families CAN, Southwest Human Development,
TERROS, and Crisis Nursery. Four caseworkers and one
parent aide provide services to their clients. Child and
Family Resources performed assessments on 35 percent
of all families referred to them and provided service plans
to 33 percent of all families referred to them.
n Marana Unified School District—Marana Unified
School District serves families in northwestern Pima
County. They have 13 collaborators, including the Ma-rana
Health Center, Rural Domestic Violence Advocate,
Picture Rocks Community Center, Rural Health Partner-ship,
Northwest Interfaith Council, Picture Rocks Beauty
and Barber, and the Salvation Army. One caseworker and
one program supervisor provide services to their clients.
Marana Unified School District completed assessments
for 50 percent of all families referred to them and pro-vided
service plans to 43 percent of all families referred to
them.
n Our Town Family Center—Our Town Family Center
serves families in southeastern Pima County. They have
41 collaborators, including Fred Acosta Job Corps, City of
Tucson, Tucson Metropolitan Ministry, Pima Community
College Center for Training and Development, and Tuc-son
Center for Women and Children. Fourteen workers
conduct assessments and provide services to their clients.
Statutory Evaluation Components
42
OFFICE OF THE AUDITOR GENERAL
Our Town Family Center performed assessments on 37
percent of all families referred to them and provided
service plans to 30 percent of all families referred to them.
B.7. Recommendations regarding program administration.
As discussed in Finding I (see pages 13 through 17), Finding II
(see pages 19 through 27), and Finding III (see pages 29
through 33), DES should improve its oversight of the program
by:
1. Revising its quarterly monitoring process to include:
a. a review of costs reported in the database compared to
those documented in the file;
b. a review of the number and type of services reported in
the database to those recorded in the files and ensuring
that providers document the reasons why identified
service needs are not met;
c. a review of the demographic information reported in
the database compared with that collected in the file to
ensure that accurate and complete demographics in-formation
is collected on families served by the pro-gram;
d. a review of provider collection of the Brief Family As-sessment
Scale and following up with providers that
have low completion rates; and
e. a review of the recruitment, activity, and involvement
of Local Advisory Boards and guidance to both pro-viders
and the Boards as needed.
2. Revising and developing documentation to further assist
providers in reporting accurate information regarding the
program as required in provider contracts, training, and
existing forms. Specifically, DES should:
a. develop a manual for reporting cost to ensure that pro-viders
collect and report accurately the cost of services;
Statutory Evaluation Components
43
OFFICE OF THE AUDITOR GENERAL
b. modify billing forms and the database to ensure that all
provider costs are captured; specifically, referral and
overhead costs; and
c. develop a user’s manual explaining how providers
should collect the required demographics informa-tion.
3. Reviewing reasons that identified service needs are not
being provided and address those concerns through de-veloping
service availability or training.
4. Using the improved service and cost information to de-termine
the appropriateness of capitated reimbursement
rates so that new rates can be set during the fiscal year
2001 contract negotiation, if appropriate.
5. Providing additional training to providers on the use of
the Brief Family Assessment Scale as a case management
tool and encouraging families to complete the assess-ment.
B.8. Information concerning the extent to which goals and
objectives of the pilot project are being successfully
met.
DES established one goal for the program: to enhance par-ents’
ability to create safe, stable, and nurturing home envi-ronments
that promote safety of all family members and
healthy child development. They also established two objec-tives
for the program, which are: 1) to increase safety of chil-dren
in their family home, and 2) to increase parenting com-petence
or effectiveness. As discussed in Finding I (see pages
13 through 17), the program data does not suggest that the
program has had an effect on home safety yet. There were no
meaningful differences between the CPS report rates of par-ticipant
and comparison group families. However, Pilot Pro-gram
families evidenced a modest though statistically signifi-cant
reduction in their risk for child abuse and neglect as re-ported
in the worker-completed Family Risk Scale. The fam-ily-
completed Brief Family Assessment Scale results are in-conclusive
because they are not representative of all families
served.
Statutory Evaluation Components
44
OFFICE OF THE AUDITOR GENERAL
B.9. Information on the effect of the pilot program in reducing
the number of children placed outside of the home and
in reducing the number of reports to child protective
services and the number of investigations by child pro-tective
services.
While CPS reports and investigations have declined since the
Pilot Program’s implementation, the cause of these declines
cannot be directly attributed to the program. In addition, the
number of out-of-home placements has increased since the
program’s implementation. However, it may not be reason-able
to expect changes to broad system-wide issues in such a
short time and with such limited application of the program.
Because fewer reports were received and many of the low-and
potential-risk reports were referred to the Pilot Program,
the total number of investigations performed by DES since the
program’s implementation has declined. As shown in Figure 3
(page 45), the total number of reports CPS has received has de-clined
in both Pilot Program counties and in counties that did
not have the Pilot Program. In fact, the rate of decline has been
proportionately steeper in counties that did not have the pro-gram.
Therefore, it does not appear that the program is di-rectly
responsible for the decline in reports.
However, DES has reported improved response rates to all
CPS reports since the program’s implementation. For example,
in the month prior to the Family Builders implementation,
DES had an 80 percent investigation rate for low-risk reports
in Maricopa and Pima Counties and a 67 percent investigation
rate for potential-risk reports received in Maricopa and Pima
Counties. Since the first month of the implementation, DES
has reported that 100 percent of all reports received in Mari-copa
and Pima Counties were either investigated by DES or
referred to the Family Builders Pilot Program.
In addition, there is no difference between the rate of out-of-home
placements for Pilot Program and non-Pilot Program
families. Specifically, approximately 1 percent of families who
received services from the Pilot Program and families who de-clined
services from the program later had a child removed
from their home.
Statutory Evaluation Components
45
OFFICE OF THE AUDITOR GENERAL
Moreover, the total number of children in out-of-home place-ment
has increased over the last three fiscal years with the
major portion of the increase occurring within the first two
years. At the end of fiscal year 1997, the State was providing
out-of-home placement services to 6,065 children. The Pilot
Program was introduced midway through fiscal year 1998
and the State was providing out-of-home placement services
to 6,708 children by the end of that year. By the end of fiscal
year 1999, DES reported that 6,734 children received out-of-home
care.
Figure 3
Family Builders Pilot Program
Reported Instances of Child Abuse and Neglect1
Years Ended June 30, 1996 through 1999
0
5,000
10,000
15,000
20,000
25,000
1996 1997 1998 1999
Maricopa County Pima County Other Counties
1 The Family Builders Pilot Program was available in Maricopa and Pima Counties only after January
1, 1998. The program was not available to the other counties at that time.
Source: Auditor General staff analysis of Child Protective Services monthly report data from July 1,
1996 through June 30, 1999.
Family Builders Pilot Program Implemented
Number of Reports
Statutory Evaluation Components
46
OFFICE OF THE AUDITOR GENERAL
B.10. Information on a comparison of outcomes for families
served by the Pilot Program with families reported to
Child Protective Services in the Department of Economic
Security in other locations.
Finding I (see pages 13 through 17) explains that outcomes
for the Pilot Program participants were no different than
those of families served by DES in other locations. Specifi-cally,
there was no meaningful difference in the proportion of
participants who later received a CPS report and the propor-tion
of families from either comparison group who subse-quently
received reports. In addition, there was no significant
difference between the proportion of participant families
who have subsequent substantiated CPS reports and the
proportion of families in either comparison group who later
received a substantiated CPS report.
OFFICE OF THE AUDITOR GENERAL
APPENDIX
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
a-i
OFFICE OF THE AUDITOR GENERAL
APPENDIX
Assessment Tools
n Family Risk Scale—is completed by the Family Builders case-worker
at participants’ intake to and exit from the program. The
Family Risk Scale emphasizes “parental characteristics and fam-ily
conditions that are believed to be predictors or precursors of
child maltreatment.”1
The Family Risk Scale contains 26 individual dimensions that as-sess
the family’s adequacy and risk for the component being
measured as described in Table 7, page a-ii. A score for each di-mension
is provided by the caseworker, using either a 4-point, 5-
point, or 6-point response scale. Within these scales, “1” repre-sents
the highest degree of adequacy (and lowest degree of risk)
and higher numbers represent increasing degrees of inadequacy.
Five dimensions of the Family Risk Scale apply to the entire
family, 8 apply to the caregivers, and 13 apply to the children. In
addition, if a child in the family is also a parent, caseworkers also
rate that child on two of the parent dimensions. The Family Risk
Scale does not give one overall measurement of risk. Instead risk
is assessed in three separate summary scales: economic, parent,
and child-centered risk.
1 Magura, Stephen, Beth Silverman Moses, and Mary Ann Jones. Assessing Risk
and Measuring Change in Families: The Family Risk Scales. Washington, D.C.:
Child Welfare League of America, 1987.
Appendix
a-ii
OFFICE OF THE AUDITOR GENERAL
Table 7
Family Builders Pilot Program
Family Risk Scale Assessment Areas and Scores
Risk Assessment Area
Range of
Scores
Economic risk factors
Habitability of family residence 1-5
Suitability of living conditions 1-5
Financial problems 1-4
Physical needs of child 1-4
Parent-centered areas
Adult relationships in household 1-4
Parent’s mental health 1-5
Knowledge of child care and development 1-4
Parent’s substance abuse 1-5
Parent’s motivation for problem solving 1-5
Parent’s cooperation with agency 1-4
Preparation for parenthood 1-4
Supervision of child under age 10 1-4
Parenting of child age 10 and over 1-4
Physical punishment of child 1-5
Verbal discipline of child 1-4
Emotional care and stimulation of infant under age 2 1-4
Child-centered areas
Parent’s attitude to preventing placement 1-4
Emotional care and stimulation of child age 2 and older 1-5
Child’s mental health 1-5
Child’s school adjustment 1-6
Child’s delinquent behavior 1-6
Child’s home-related behavior 1-5
Child’s cooperation with agency (if also a parent) 1-4
Child’s preparation for parenthood (if also a parent) 1-4
Source: The Family Risk Scales developed by the Child Welfare League of America and
published by Stephen Magura, Beth Silverman Moses, and Mary Ann Jones in As-sessing
Risk and Measuring Change in Families: The Family Risk Scales, Washington,
D.C.: Child Welfare League of America, 1987.
n Brief Family Assessment Scale (BFAS)— is a paper-and-pencil
assessment completed by participant families at intake to
Appendix
a-iii
OFFICE OF THE AUDITOR GENERAL
and exit from the program. The BFAS measures the family’s per-ception
of their strengths, resources, and problems in the fol-lowing
seven areas:
n Personal Stress
n Family Support
n Economic Stress
n Aggressive Behavior
n Problems with My Child
n Drug Use
n Alcohol Use
The seven subscales are comprised of 76 individual statements. The
family is asked to indicate how often they experience the situation
described in the statement using the following scale:
1 None of the time
2 Very rarely
3 A little of the time
4 Some of the time
5 A good part of the time
6 Most of the time
7 All of the time
X Does not apply
For questions that apply to only one family member, families are
asked to respond based on the family member to whom the item
applies the most. For instance, if the person completing the assess-ment
never becomes hostile and threatening, but another family
member exhibits that behavior a good part of the time, then the per-son
should respond with a 5 to the statement related to that issue.
Statistical Analysis of
Subsequent CPS Report Rates
n Analysis of All Families—As reported in Finding I (see
pages 13 through 17), the review of CPS data from July 1,
1995 to June 30, 1999, found no meaningful difference be-tween
the proportion of participant families who received
Appendix
a-iv
OFFICE OF THE AUDITOR GENERAL
CPS reports after the receipt of the initial, referring report
and the proportion of families from either comparison group
who subsequently received reports.
To further explain, the Chi-Square test of significance showed
that there was a statistically significant difference between the
proportion of participant families with subsequent CPS reports
(31.9 percent) and the proportion of families who did not re-ceive
Family Builders services (24.2 percent). The Chi-Square
result was 26.86 (1, N=4,249), p< .001. However, the intensity of
the relationship between being in Family Builders and receiv-ing
a CPS report was so weak (f = -.08) that there is no mean-ingful
or practical significance in the resulting Chi-Square
value.
The same analysis was performed to determine differences
between the proportion of participant families with subsequent
CPS reports (31.9 percent) and the proportion of families served
by DES in other locations with subsequent CPS reports (28.5
percent) and found a similar result. Specifically, the Chi Square
result was 5.26 (1, N = 4,778), p<.022 and the Phi was -.033.
Again, we have statistical significance with a weak association
resulting in no meaningful difference.
n Analysis of Families with No Prior Reports—As reported
in Finding I (see pages 13 through 17), when looking only at
families whose first involvement with CPS during the period
examined was through the referring CPS report, the analysis
showed that there is no meaningful difference between the
proportion of Pilot Program families and the proportion of
two different comparison group families who had a subse-quent
CPS report.
Specifically, the Chi-Square test of significance showed that
there was no statistically significant difference in the proportion
of participant families with subsequent CPS reports (25.0 per-cent)
and the proportion of families served by DES in other
counties who later received a CPS report (23.6 percent). Specifi-cally,
the Chi Square result was .697 (1, N = 3,266), p < .404 and
the Phi was -.015.
Appendix
a-v
OFFICE OF THE AUDITOR GENERAL
The Chi-Square test of significance showed that there was a
statistically significant difference between the proportion of
participant families with subsequent CPS reports (25.0 percent)
and the proportion of families who did not receive Family
Builders services and later received CPS reports (17.9 percent).
The Chi-Square result was 18.43 (1, N = 3,022) p < .001; how-ever,
the relationship was weak (f = -.079). Therefore, there is
no meaningful or practical significance in the resulting Chi-
Square value.
n Analysis of families excluding those referred back to CPS
by Family Builders staff—In a separate analysis, evaluation
staff excluded families whose Family Builders case was closed
because the family was referred back to CPS. There was no
meaningful difference in the proportion of families with subse-quent
CPS reports or the proportion of families with subse-quent
substantiated CPS reports between the participants and
either comparison group.
n Analysis of demographic differences—As reported in the
Introduction and Background (see pages 1 through 12), some
statistically significant demographic differences exist between
the participants and comparison groups. Evaluation staff per-formed
a logistic regression analysis to determine the impact of
ethnicity, gender, and age on subsequent CPS report rates for
families in all three groups. Only one factor, ethnicity, had an
impact. White caregivers in all groups were more likely to have
a subsequent CPS report. However, White caregivers are un-derreported
in the Family Builders’ participant group relative
to one comparison group and proportionately similar to the
second comparison group. Based on this analysis, it is unlikely
that differences in ethnic composition of the groups resulted in
Family Builders participants having a higher rate of reports.
n Analysis of CPS report priority of originating report—Prior-ity
three and four reports are eligible for the Family Builders
program, but priority three reports involve more serious alle-gations
of child abuse and neglect. The distribution of priority
three and four reports among participant families and those
families who were referred, but did not receive, services were
Appendix
a-vi
OFFICE OF THE AUDITOR GENERAL
the same. Families in the counties without the Pilot Program
had significantly more priority three reports than participant
families did Chi Square 652.43. (1, n = 4,778), (p < .001, f =.370)
a-vii
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
OFFICE OF THE AUDITOR GENERAL
AGENCY RESPONSE
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
Jane Dee Hull
Governor
ARIZONA DEPARTMENT OF ECONOMIC SECURITY
1717 W. Jefferson, P.O. Box 6123, Phoenix, AZ 85005
John L. Clayton
Director
Ms. Debbie Davenport, CPA
Office of the Auditor General
2910 North 44th Street, Suite 410
Phoenix, Arizona 85005
Dear Ms. Davenport:
The Department wishes to thank the Office of the Auditor General for the opportunity to respond to the
recently completed audit of the Family Builders program.
The Department is proud to have created a program that is starting to show some positive outcomes in
such a short period of time. We have a 100 percent response rate to reports of child abuse and neglect
utilizing a combination of investigations by CPS and referrals to Family Builders. The Family Risk Scale,
which is completed at intake and exit, showed statistically significant reductions in risk on all three
subscales. The Brief Family Assessment Scale showed a statistically significant difference on two of the
seven subscales between the intake and exit scores.
The Department developed a client database to aid in the record keeping and evaluation of the program.
Of the families who responded to a client satisfaction survey, 91 percent felt their family was somewhat
or a great deal better off as a result of their involvement with Family Builders. In addition, 92 percent
reported feeling they were a great deal or somewhat better able to deal with their problems as a result of
the Family Builders program.
Creating a new program is not without its challenges. We welcome the constructive suggestions for
strengthening this important program. As programmatic issues have been raised, the Department
responded by addressing the issues in new worker training, during the quarterly site monitoring visits, and
at the quarterly partnership meetings with the providers.
The Family Builders program has brought the community and CPS together in a partnership to provide
services to families and to prevent subsequent child abuse and neglect of children. It has been
recognized at the national level as an innovative program serving the children and families of Arizona,
winning the 1999 Innovations Award from the Council of State Governments.
Page 2
Debbie Davenport
The recommendations pertaining to each finding will be implemented as discussed in our accompanying
response. We would like to express our appreciation for the time and effort that you have invested in
this important evaluation.
If you have any questions or need additional information please contact Anna Arnold, Acting Assistant
Director at 542-3598, or me at 542-5678.
Sincerely,
John L. Clayton
Enclosure
DEPARTMENT OF ECONOMIC SECURITY
RESPONSE TO THE
FAMILY BUILDERS PROGRAM EVALUATION
FINDING I: Family Builders Unable to Show Impact on Child Safety at This Time
The Department agrees with the Auditor General that the Family Risk Scale assessment results indicate
that program families experienced a statistically significant decrease in risk in each of the three measured
areas: economic; parent centered and child-centered. Further, for the families who completed the
voluntary Brief Family Assessment Scale, the differences between intake and exit scores were
statistically significant for two of the seven subscales. On both of these scales, the risk for child abuse
and neglect decreased.
Given the positive results listed above, the Department believes the finding should read as follows:
Family Builders Ability to Show Impact on Child Safety is Inconclusive at this Time. Therefore, the
finding of the Auditor General is not agreed to, but the recommendation which states DES should ensure
that accurate and complete program information, such as service delivery data, demographic data, and
family risk assessments, is collected as specified in provider contracts, training, and existing forms, to
ensure that program outcomes can be fully measured in the future is agreed to and the audit
recommendation will be implemented. The Department has already implemented all recommended
changes and remains committed to ensuring that program outcomes are fully measured.
DEPARTMENT OF ECONOMIC SECURITY
RESPONSE TO THE
FAMILY BUILDERS PROGRAM EVALUATION
FINDING II: DES Needs to Examine Costs and Number of Services Delivered
The Department does not agree with the statement made by the Auditor General that says the providers
are overpaid an average of $1,700. During fiscal year 1999, the Department’s review of costs shows
the average rates paid to Family Builders providers were $93 for a referral, $266 for a completed
assessment, and $2,855 for a completed service plan. An analysis of the provider’s reported costs
showed that the average cost for the providers to respond to a referral and complete an assessment was
$383, a net cost to the provider of $24 per referral and assessment completed. The average cost for a
family that went through the referral, assessment, and service plan was $2,964; while the average
reimbursement to the provider was $3,214. This results in a net profit of approximately $250. With
only 28% of the reports referred to the Family Builders providers completing a service plan, the
provider has the potential of having a net loss depending on the number of referrals and assessments
completed.
The finding of the Auditor General is agreed to and the following recommendations will be implemented.
To ensure appropriate reimbursement rates and service levels, DES should:
1. Modify its billing forms and database to ensure that all provider costs are captured; specifically,
referral and overhead costs. The majority of the forms have already been modified and
will more accurately capture the necessary information.
2. Develop a manual for reporting costs to ensure providers accurately collect and report costs.
The Department, will develop an instructional manual which will ensure providers
accurately collect and report costs and will provide training and technical assistance to
providers in this process.
3. Include the following procedures in its quarterly monitoring process:
A) Verify that the type, number, and costs of services reported in the database
reflect what is documented in the file; and
B) Ensure that providers’ files adequately explain the reasons identified service
needs are not met. The Department modified program forms in June 1999,
and starting in July 1999, Family Builders Specialists review the case files
and gather information on the type, number and cost of services reported.
This information is compared to the database and any discrepancies are
noted in the site monitoring report which is given to the provider. The
provider is responsible for correcting the discrepancies. Further, at site
monitoring visits service plans are reviewed, specifically the portion that
discusses whether goals have been accomplished and reasons service
needs are not being met. The results are shared with the provider who is
responsible to ensure staff are completing the forms accurately and
completely and families are receiving needed services.
4. Examine the reasons that identified service needs are not being provided and address those
concerns through developing service availability or training. The Family Builders Specialists
started providing additional training on a quarterly basis on how to identify needs, locate
services and develop resources.
5. Use the improved service and cost information to regularly review the appropriateness of
capitated reimbursement rates so that new rates can be set during the fiscal year 2001 contract
negotiation, if appropriate. The Department will gather more accurate and detailed cost
information and adjust the reimbursement rates during the next Request for Proposal
process, if appropriate.
DEPARTMENT OF ECONOMIC SECURITY
RESPONSE TO THE
FAMILY BUILDERS PROGRAM EVALUATION
FINDING III: DES Should Improve Program Oversight to Ensure All Requirements Are Met
Creating a new program is not without its challenges and as programmatic issues are raised, the
Department responds by addressing the concern in new worker training, during the quarterly site
monitoring visits of providers, and at the quarterly partnership meetings of providers. The Department is
committed to ensuring providers fulfill the requirements of their contracts, so in turn, program families
receive appropriate services. These services are provided to families to assist them in creating a safe,
stable and nurturing home environment that promotes safety of all family members and healthy child
development.
The finding of the Auditor General is agreed to and the following audit recommendations will be
implemented.
1. To ensure that accurate and complete demographics information is collected on program
families, DES should:
A) Develop a user’s manual that details data collection procedures; and
B) Compare the information reported in the database with information collected
from file reviews during the quarterly review process. The Department is
already comparing information reported in the database against
information collected from file reviews, and will continue to do so. A
users manual will be created that gives detailed instructions on how to
accurately report demographic information.
2. To improve usage of the BFAS, DES should:
A) Give additional training to providers on using the BFAS as a case management
tool and on encouraging families to complete the assessement; and
B) Monitor the collection and use of the BFAS as part of the quarterly review
process. The Department continues to stress to the providers the
importance of BFAS completion. Detailed training on how to complete
the BFAS and ways to encourage family participation is provided during
provider new employee training. Since October 1999, the Department has
been reviewing the case files and provider database in an effort to ensure
the BFAS is completed and entered into the database. Any discrepancies
are given to the provider in their site monitoring report. A policy was
adopted in January 2000 which states the provider staff will complete the
post-BFAS when it appears the family is disengaging from the program
rather than wait until the families leaves the program.
3. To ensure that Local Advisory Boards fulfill their statutory and contractual requirements, DES
should conduct reviews of Board activities and provide assistance with recruiting and involving
Board members. The Family Builders Specialists will meet with the Local Advisory
Boards and provide technical assistance, written material and training to ensure the
Boards are fulfilling their statutory and contractual requirements.
Other Performance Audit Reports Issued Within
the Last 12 Months
99-5 Department of Gaming
99-6 Department of Health Services—
Emergency Medical Services
99-7 Arizona Drug and Gang Policy
Council
99-8 Department of Water Resources
99-9 Department of Health Services—
Arizona State Hospital
99-10 Residential Utility Consumer
Office/Residential Utility
Consumer Board
99-11 Department of Economic Security—
Child Support Enforcement
99-12 Department of Health Services—
Division of Behavioral Health
Services
99-13 Board of Psychologist Examiners
99-14 Arizona Council for the Hearing
Impaired
99-15 Arizona Board of Dental Examiners
99-16 Department of Building and
Fire Safety
99-17 Department of Health Services’
Tobacco Education and Prevention
Program
99-18 Department of Health Services—
Bureau of Epidemiology and
Disease Control Services
99-19 Department of Health Services—
Sunset Factors
99-20 Arizona State Board of Accountancy
99-21 Department of Environmental
Quality—Aquifer Protection Permit
Program, Water Quality Assurance
Revolving Fund Program, and
Underground Storage Tank Program
99-22 Arizona Department of Transportation
A+B Bidding
00-1 Healthy Families Program
00-2 Behavioral Health Services—
Interagency Coordination of Services
00-3 Arizona’s Family Literacy Program
Future Performance Audit Reports
Department of Public Safety—Aviation Division
Department of Agriculture’s
Animal Disease, Ownership and Welfare Protection Program