Annual evaluation, Healthy Families Pilot Program

              ANNUAL EVALUATION
HEALTHY FAMILIES
PILOT PROGRAM
State o f Arizona of fice
of the
Auditor General
PERFORMANCE AUDIT
Report to the Arizona ~ egislature
By ~ ouglasR . Norton
Auditor General
~ ovemLer1 996
Report 96- 11
DOUGLAS R. NORTON, CPA
AUDITOR GENERnL
STATE OF ARIZONA
OFFICE OF THE
AUDITOR GENERAL
DEBRA K. DAVENPORT, CPA
DEPUTY fi" D, TOR GENERAL
November 19,1996
Members of the Arizona Legislature
The Honorable Fife Symington, Governor
Dr. Linda J. Blessing, Director
Department of Economic Security
Transmitted herewith is a report of the Auditor General, an Annual Evaluation of the Healthy
Families Pilot Program. This report is in response to the provisions of Laws 1994, Ninth S. S.,
Ch. 1, 59.
This is the second in a series of reports to be issued on the Healthy Families Pilot Program.
The report addresses the families served by the Program and the types of problems they face,
as well as the fact that families are accessing fewer services than the Program is designed to
provide. The report also includes a discussion of statutory annual evaluation components
including the Program's attrition rate and cost per family.
My staff and I will be pleased to discuss or clarify items in the report
This report will be released to the public on November 20,1996.
Sincerely,
~ o u uRs. N orton
Auditor General
Enclosure
2 9 1 0 NORTH 44TH STREET 1 SUITE 4 1 0 . PHOENIX, ARIZONA 8 5 0 1 8 . ( 6 0 2 ) 5 5 3 - 0 3 3 3 . FAX ( 6 0 2 ) 5 5 3 - 0 0 5 1
SUMMARY
The Office of the Auditor General has completed the second year of a three- year evaluation
of the Healthy Families Pilot Program. This evaluation was conducted pursuant to the
provisions of Laws 1994, Ninth S. S., Ch. 1, 59. This second- year, interim evaluation report
provides descriptive information regarding the Program. The final evaluation report will focus
on the Program's impact and is to be released on or before December 31,1997.
The Legislature established the Healthy Families Pilot Program through Laws 1994, Ninth S. S.,
Ch. 1, $ 59, also known as the Children and Family Stability Act of 1994, to address the growing
need for child abuse prevention.
The Arizona Department of Economic Security ( DES) is responsible for administering the
Healthy Families Pilot Program. The Legislature appropriated $ 1.7 million for fiscal year 1995,
and $ 3 million annually for fiscal years 1996 through 1998 to DES to implement the Healthy
Families Pilot Program. The Children and Family Stability Act provided monies that allowed
the Healthy Families Pilot Program to be implemented at 13 sites, beginning in January 1995.
DES awarded 5 contracts to serve 13 sites in 6 counties.
Healthy Families Pilot Program Serves
Families with Multiple Problems
( See pages 9 through 12)
Chents served by the Healthy Families Pilot Program typically face a variety of problems in
their lives that make rearing children difficult and increase the risk of child abuse. Most
program participants exhibit a moderate to severe risk for committing child abuse, and face
problems that include unwed motherhood, substance abuse, unstable housing, and poorly
functioning families.
The Program targets families who need its services. Most program participants exhibit a
moderate to severe risk for committing child abuse or neglect and face difficult problems.
Several stressors, including limited finances, unemployment, and marital problems, contribute
to the risks of abuse and neglect Additionally, about 16 percent of the Program's families have
a high potential for child abuse as measured by The Child Abuse Inventory.
Program participants face troubled living situations. Most participants entering the Program
are part of families that are not supportive of other family members and are not able to deal
with stress or change. Overall, approximately 70 percent of the Program's families are poorly
functioning.
Families Access Fewer
Services Than Envisioned
in Program Model
( See pages 13 through 17)
The Healthy Families Pilot Program outlines an ideal frequency and duration of services a
family should receive in order to prevent child abuse and neglect. However, many families
are not attaining these service goals. Major obstacles to service delivery include families not
being home for scheduled visits, a to- date attrition rate of 47 percent, and families' not
accessing other services to which the Program refers them. These obstacles are resulting in
considerable deviation from the program model in terms of families' length of enrollment
( duration), and frequency of visits ( intensity). For example, data show that, during their first
6 months in the Program, only 11 percent of the families were home for each scheduled visit.
On average, families did not keep 4 of 15 or more planned visits. In addition, almost half of
the families are dropping out before completing the Program. Although the Program is
designed to provide services until a child's fifth birthday, families leaving the Program are
enrolled for an average of 168 days. Finally, program workers report that many families are
not using services to which they are referred, such as immunization clinics, nutrition
programs, etc.
It is not yet clear how such deviations from the program model in terms of families receiving
fewer services for a shorter time will impact program outcomes, and whether families will
benefit from receiving fewer services than planned.
Statutory Annual Evaluation Components
( See pages 21 through 27)
This report contains responses to the nine statutory annual evaluation components in
accordance with Laws 1994, Ninth S. S., Ch. 1/ 59. A variety of issues are addressed under the
statutory section. Of particular interest is the Program's high attrition rate, with 47 percent of
those families enrolled since January 1995 already disenrolled from the Program. The high
attrition rate also contributed to a high yearly cost per family for the Program.
Table of Contents
Introduction and Background . . . . . . . . . . . . . . . . . . . . . . . .
Finding I: Healthy Families Pilot Program
Serves Families with Multiple Problems . . . . . . . . . . . . . . .
Program Targets Families
Who Need Its Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Finding II: Families Access Fewer Services
Than Envisioned in Program Model . . . . . . . . . . . . . . . . . .
Enrolled Families Do Not
Receive Program's Full
Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Families That Need
Services May Not Enroll or
Actively Participate in the Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Limited Family Participation
Impacts Program Outcome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Paae
1
Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Other Pertinent Information . . . . . . . . . . . . . . . . . . . . . . . . . . 19
National Concerns About the
Family Stress Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Statutory Annual
Evaluation Components . . . . . . . . . . . . . . . . . . . . . . . . . .
Agency Response
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . a- i
iii
Table of Contents ( Concl'd)
Page
Figures
Figure 1: Healthy Families Pilot Program
Services Provided and Percentage
of Mothers Receiving Services
January 1995 through Apd 1996 . . . . . . . . . . . . . . . . . . . . . . . . .
Figure 2: Healthy Families Pilot Program
Referral Services and Percentage of
Mothers Receiving Referrals
January 1995 through April 1996 . . . . . . . . . . . . . . . . . . . . . . . . . 5
Figure 3: Healthy Families Pilot Program
Distribution of Level of Risk
Mothers and Fathers
January 1995 through April 1996 . . . . . . . . . . . . . . . . . . . . . . . . .
Figure 4: Healthy Families Pilot Program
Distribution of Expenditures
Year Ended June 30,1996
( Unaudited) ........................................* 24
Tables
Table 1: Healthy Families Pilot Program
Revenues and Expenditures by Contractor
Year Ended June 30,1996
( Unaudited) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Table 2: Healthy Families Pilot Program
Cost per Family
Years Ended June 30,1995 and 1996 . . . . . . . . . . . . . . . . . . . . . . . 26
INTRODUCTION AND BACKGROUND
The Office of the Auditor General has completed the second year of a three- year evaluation
of the Healthy Families Pilot Program. This evaluation was conducted pursuant to the
provisions of Laws 1994, Ninth S. S., Ch. 1, 59. This second- year, interim evaluation report
provides descriptive information regarding the Program. The final evaluation report will focus
on the Program's impact and is to be released on or before December 31,1997.
Legislation, Appropriations,
and Program Administration
The Legislature established the Healthy Families Pilot Program through Laws 1994, Ninth S. S.,
Ch. 1, 59, also known as the Children and Family Stability Act of 1994, to address the growing
need for child abuse prevention.
The Arizona Department of Economic Security ( DES) is responsible for administering the
Healthy Families Pilot Program. The Legislature appropriated $ 1.7 million for fiscal year 1995
and $ 3 million annually for fiscal years 1996 through 1998 to DES to implement the Healthy
Families Pilot Program. The Children and Family Stability Act provided monies that allowed
the Healthy Families Pilot Program to be implemented at 13 sites beginning in January 1995.'
DES awarded 5 contracts to serve 13 sites in 6 counties, as follows:
Tucson Association for Child Care received two separate contracts for urban and rural
sites, and manages three urban sites in Pima County and three rural service sites in
Cochise and Santa Cruz Counties. Its urban contract also includes the cost of providing
statewide program oversight through a quality assurance coordinator.
Southwest Human Development was awarded a contract for managing four sites in
Maricopa County.
Marcus J. Lawrence Medical Center serves one site in Yavapai County.
Coconino County Department of Public Health has two sites, both serving Coconino
County.
1 Prior to the legislatively mandated Pilot Programl DES administered a Healthy Families Program though its
Child Abuse Prevention Fund. Currently, five additional sites are in operation through the use of the Child
Abuse Prevention Fund and other monies.
In addition to these site contracts, DES contracted out the database management function to
a data management firm.
Background
Child abuse has become an increasingly serious problem in the United States. Between 1976
and 1994, the reported number of child abuse and neglect cases increased more than 4 times,
from 669,000 to over 3 million. In 1976, about 10 out of every 1,000 American children were
reported to have been abused or neglected. By 1994, this ratio increased nearly 5 times, to 47.
Arizona experienced similar increases in reported child abuse between 1984 and 1994. For
example, the number of cases involving child abuse almost doubled; however, the number of
reported cases dipped 7 percent in fiscal year 1994. It is not clear from the available data
whether the drop indicates a new trend or an exception. Based on the nationwide data/ it is
safe to assume that the upward trend continues. Additionally, the number of cases involving
child abuse and neglect- related fatalities in Arizona almost doubled between 1991 and 1994.
Program Model
Arizona's Healthy Famhes Pilot Program uses a home visit model based on Hawaii's Healthy
Start Program, a nationally recogruzed child abuse and neglect prevention model. The Healthy
Families Pilot Program is a community- based program serving families with newborns, which
aims to reduce stress, enhance family functioning, promote child development, and minimize
the incidence of abuse and neglect. The Program has 14 critical elements. Some key elements
of the Program follow:
Participation is voluntary;
Systematic screening determines families' eligibility;
Services are based on each family's individual needs and risk level;
r Collaborative, community- based services are integral to the program; and,
Families are linked to health care systems.
The Healthy Families Pilot Program providers target an area for service if it is in a zip code
zone in which a high rate of child abuse has occurred and if this zone is within the grovidefs
service area. Families living in those areas can enroll in the Program if needs assessments,
conducted by family assessment workers, reveal a risk of child abuse and families agree to
participate. Initially, services are provided by family support specialists during weekly home
visits. Home visits are less frequent as families progress through the Program, which is
designed to serve families until the child's fifth birthday.
Program Goals
and Services
The Legislature specified the following five goals for the Healthy Families Pilot program. A
range of services and referrals are provided to participants to ensure the Program meets these
five basic goals.
Goal One: Reduce child abuse and neglect- The Program provides an assortment of
services to meet this goal, including education on chld development, nutrition, support
groups, modeling appropriate behavior, life coping skills, play groups, emotional support,
and crisis management and intervention. Additionally, workers may refer families to social
service and mental health agencies.
Goal Two: Promote child wellness and proper development- To meet this goal, family
support specialists provide information on child development and child health care, such
as immunizations and the importance of well- baby visits. Workers may also give families
transportation to hospital and doctor appointments.
Goals Three and Four: Strengthen family relations and promote family unity- To meet
these goals, family support specialists work to draw fathers/ spouses into the Program.
They provide social events and support groups where men can meet and discuss issues
relevant to families, relationships, and parenting. In addition, family support specialists
try to get fathers involved in the home visits. Family support specialists also work with
mothers by modeling appropriate behavior, building and improving communication skills,
increasing self- esteem, and respecting the client
Goal Five: Reduce dependency on drugs and alcohol- This goal is addressed by
referring the family member or the entire family to substance abuse counseling services.
The family support specialists will also provide information on the dangers of substance
abuse.
Many of these services were mandated by the Family Stability Act These mandated services
include nutrition and health education, assistance in obtaining childhood immunizations,
assistance in obtaining private and public financial assistance when necessary, and help with
developing parenting and coping skills. The Healthy Families Pilot Program's service
distribution and referrals are illustrated in Figure 1 ( see page 4), and Figure 2 ( see gage 5).
Figure 1
Healthy Families Pilot Program
Services Provided and Percentage of
Mothers Receiving Services
January 1995 through April 1996
mEducation on Child Development EiAssist. with Mant/ Mother Bonding & Attachment
LJE motional Support ~ Mormationan d Referral
OLife Coping UTransportation
Number of Families = 1,045
Source: Auditor General staff analysis of Healthy Families database.
Figure 2
Healthy Families Pilot Program
Referral Services and Percentage of
Mothers Receiving Referrals
January 1995 through April 1996
Referral Services Offered
mWomen, Infant and ChildrenJFood Stamps EMedical Services
EJAid to Families with Dependent ChildrenIGen'l Assistance UTransportation
rnSocial Services UEducation
EUFinancial Support = Child Care
EflJob Services rmMental Health Services
IDrug/ Alcohol Treatment
Number of Families = 1,045
Source: Auditor General staff analysis of Healthy Families database.
Scope and Methodology
The Family Stability Act requires the Office of the Auditor General to annually evaluate the
Healthy Families Pilot Program's results. These evaluations focus only on those program sites
funded by this Act Those Healthy Family Arizona sites funded by the DES Child Abuse
Prevention Fund are evaluated by a separate contractor. The Family Stabihty Act requires
evaluating the Program's effectiveness, the level and scope of program services, eligibility
requirements, and the number and demographic characteristics of program participants.
A variety of methods were used during this evaluation. Interviews and two focus groups were
conducted with DES and Healthy Families Pilot Program staff. We also analyzed program
revenues and expenditures and program enrollment information. Additionally, we reviewed
family characteristic assessment tools, reviewed files of 36 participants, conducted literature
reviews, and directly observed the Program through 10 site visits, 2 staff training seminars,
and 8 home visits with program participants.
Six assessment tools are used to collect information about the families who are served by the
Healthy Families Pilot Program. A description of the tools used is included in the Appendix
( see pages a- i through a- ii). The assessments used include the Family Stress Checklist, Child
Abuse Potential Inventory, and FACES 11, which is a checklist that measures family
functioning. To determine a participant's level of risk for committing child abuse or neglect,
information from the Family Stress Checklist and the Child Abuse Potential Inventory was
analyzed. A family's functioning level was obtained by analyzing data from the FACES II. In
addition to these formal assessments, we are tracking the progress of three program families
through home visits every three months. Finally, information was requested from Healthy
Famhes programs in 15 other states. Limited information was available from evaluations of
Healthy Families programs in eight states. Seven Healthy Families programs had not
completed data collection and analysis, and were unable to provide any data at this time.
The assessments are used for a variety of purposes. The Family Stress Checklist is used to
screen families for program eligibility. If this instrument indicates that either the mother or
father of the baby has a moderate or severe potential for committing abuse or neglect, the
family is eligible for the Program. The other assessments are used for program evaluation
purposes only.
In our first- year evaluation, we reported that DES awarded contracts in a timely and efficient
manner, developed participant eligibility criteria as mandated by the Legislature, and
operated with low administrative costs. However, the report also noted that DES needed to
improve its contract management practices. DES has since improved its contract management
practices and the= are no problems to report for the second year. Therefore, this second- year
report provides information specific to the Program's implementation. Specifically, it presents:
Information on the types of problems faced by the Healthy Families Pilot Program
participants;
Information on participants' use of sewice referrals; and
A report on statutory annual evaluation components including client characteristics and
program costs.
Our third and final report will focus on the Program's effectiveness in meeting its goals and
objectives, and the Program's impact on its participants. This kind of information is not
contained in this second report because there are not enough families who have been in the
Program long enough to evaluate its outcome measures. In addition, we have not yet finished
collecting information on the comparison group. 1
The Auditor General and staff express appreciation to the Director of the Department of
Economic Security, the Healthy Families Pilot Program Coordinator, and the staff of DES
Division of Children and Family Services, as well as the Healthy Families Pilot Program staff
for their cooperation and assistance during the second year of the Healthy Families Pilot
Program Evaluation.
4 1 The evaluation plan has established a comparison group of families who are eligible for the Program, but have
not enrolled.
( This Page Intentionally Left Blank)
FINDING I
HEALTHY FAMILIES PILOT PROGRAM
SERVES FAMILIES WITH
MULTIPLE PROBLEMS
The Healthy Families Pilot Program serves clients who typically face a variety of problems
and stressors in their lives that make rearing children difficult and increase the risk of child
abuse. Most program participants exhibit a moderate to severe risk for committing child
abuse, and face problems that include unwed motherhood, substance abuse, unstable housing,
and poorly functioning families. In addition, families often face troubled living situations.
Healthy Families participants in other states experience similar problems.
Program Targets Families
Who Need Its Services
The Healthy Families Pilot Program is serving clients likely to commit child abuse. Various
stressors, including limited finances, unemployment, and marital problems, contribute to a
parent's risk of committing child abuse or neglect and most participants in the Healthy
Families Pilot Program experience these stressors. Many participants in the Program face
difficult problems, such as a history of drugs or sexual abuse, poor self- esteem, or chronic
unemployment. Arizona participants in general compare to participants in other states.
Stresson contribute most to families' risk of abuse and neglect- Of ten main factors that
contribute to a parent's risk of committing child abuse or neglect, stressors are the most
frequently occurring. The most consistently severe problems for mothers are stressors ( such
as limited finances, frequent job changes, and constant fighting with spouse); childhood
history ( including receiving several beatings, being raised by more than two families, history
of sexual abuse); and poor self- esteem ( for example, not close to family, no lifelines, chronic
unemployment, no prenatal care). The most consistently severe problems for fathers are
stressors ( such as limited finances, frequent job changes, and constant fighting with spouse);
childhood history ( for example, receiving several beatings, being raised by more than two
families, history of sexual abuse); and a history of substance abuse, mental health problems,
and/ or a criminal history.
Most fartzilies have moderate or severe risk- As illustrated in Figure 3 ( see gage lo), almost
all of the mothers and fathers in the Healthy Families Pilot Program have a severe or moderate
risk of chdd abuse or neglect. In only 8 percent of the families do both parents have a severe
Figure 3
Healthy Families Pilot Program
Distribution of Level of Risk
Severe
35.4%
Severe
44.3%
Moderate
51.5%
Number of Mothers= 1,010 Number of Fathers= 362
Source: Auditor General staff analysis of data provided by the Department of Economic !% mrity.
risk of abusing their children and in only 10 percent do both parents have a moderate risk of
abusing or neglecting their children. However, the remaining 82 percent of families have one
parent who is at severe risk and one who is at moderate or low risk.
In addition, about 16 percent of the Program's families have characteristics which rate them
as having a high potential for child abuse as measured by The Child Abuse Potential
Inventory. The Inventory directly measures the extent to which participants have characteris-tics
similar to child abusers. These characteristics include a low self- esteem, poor self- image,
feelings of inferiority, and feelings of guilt Additional characteristics include a tendency to
be withdrawn and socially isolated. These individuals also tend to be immature, overreactive,
irritable, self- centered, moody, lonely, and frustrated.
Families with sevm m modevate risk face dificult problems - A family with a severe risk of
chdd abuse faces more serious problems than a family with moderate risk. Case Example One
presents a description of a Healthy Families Pilot Program mother who is at severe risk for
committing child abuse due to her family history, prior drug use, and unstable financial
situation, along with her baby's health problems.
Case Example l- The mother is 22 years old and has one child. She is single but was
living with her boyfriend when the baby was born; however, her boyfriend is not the
baby's father. Her baby was born with a heart condition. The mother receives financial
assistance from Aid to Families with Dependent Children, and is also enrolled in the
Women, Infant, and Children nutritional assistance program, and the Arizona Health Care
Cost Containment System. She did not graduate from high school but has received her
GED. She has no car. The mother used drugs such as marijuana, crystal methamphet-amine,
and cocaine before the pregnancy. The baby's father uses drugs and alcohol. In
addition, the participant's father was abusive to her mother, who had a drinking problem
and died when the participant was 16 years old.
Case examples 2 and 3 illustrate families who are at moderate risk of committing child abuse.
Case example 2 is a moderate risk due to late prenatal care, lack of formal education, and
current living situation. Case Example 3 is at moderate risk due to her age, childhood history,
which included physical disciplining, lack of formal education, and lack of transportation.
Case Example 2- This mother is a nineteen- year- old single woman, with two children:
a newborn and a two- year- old. She currently receives assistance from the Arizona Health
Care Cost Containment System, but did not receive prenatal care until late in her
pregnancy. This mother has not completed high school and currently lives with her
mother. The father of the baby was shocked about the pregnancy, did not want the baby,
and is not involved in the family's life,
Case Example 3- This mother is a 17- year- old single woman. She currently lives with her
boyfriend, who is the father of her baby. The father is 19 years old, works, and goes to
school. The mother is not a high school graduate, but is interested in obtaining a GED.
This participant currently has no transportation. As a child, she was disciplined with a
belt.
These case examples illustrate Healthy Families Pilot Program participants' troubled living
situations. Healthy Families Pilot Program participants are not members of families that are
supportive of other family members, and are part of families who are not able to deal with
stress and change. Analysis of FACES I1 data reveals that approximately 35 percent of the
families do not have a cohesive structure. Specifically, within these families, individuals tend
to be disconnected from each other. In addition, approximately 25 percent of the participants
are in inflexible families who are not able to adequately adjust to stress. Overall, approxi-mately
70 percent of the Healthy Families Pilot Program families are poorly functioning. In
general, the families in the Program will have problems that, over time, will prevent healthy
family functioning.
Arizona families compare to families it2 0 t h programs- The risk level of participants in
Arizona's Healthy Families Pilot Program is comparable to the risk level of families in other
states' programs. Like Arizona, a number of other Healthy Families programs in the country
also assess participants' eligibility and risk level using the Family Stress Checklist For
example, in the Tennessee program, 98 percent of the mothers have a moderate or severe risk
level, and a smaller percentage of fathers ( 38 percent) have a moderate or severe risk level.
This compares with 96 percent of mothers and 81 percent of fathers in Arizona. This indicates
that Healthy Families Pilot Program mothers in Arizona have a comparable risk level to the
Tennessee program's mothers. This is not the case for fathers in the Arizona program. These
numbers would indicate that fathers in the Arizona program have a higher level of risk than
the Tennessee program's fathers.
It is more dficult to draw comparisons from the Chdd Abuse Potential Inventory ( Inventory).
For example, the Tennessee program uses a different criteria than is recommended by the test
developers for determining a parent's potential for committing child abuse or neglect. Using
the same criteria as Tennessee, it appears that 32 percent of Arizona's participants, rather than
16 percent, show a measured risk for committing cluld abuse. This compares to the 59 percent
of the Tennessee participants who are at- risk by Tennessee's definition. Inventory data from
states other than Tennessee was not available, because other states are not using the Inventory
or the data has not yet been collected.
There is no comparison data from other sites on the FACES 11 information at this point
FINDING II
FAMILIES ACCESS FEWER SERVICES
THAN ENVISIONED IN PROGRAM MODEL
The Healthy Families Pilot Program outlines an ideal frequency and duration of services a
family should receive in order to prevent child abuse. However, many families do not receive
the Program's full services. Many of the families in the Program do not keep appointments for
home visits and at least 40 percent of the families do not complete the Program. However, data
available from programs in other states suggests such behaviors may be typical for families
served by the Healthy Families Pilot Program. Some families who need services provided by
the Healthy Families Pilot Program may not enroll. Additionally, a high number of enrolled
families never become active in the Program. It is not yet clear how such deviations from the
program model in terms of families receiving fewer services for a shorter time may impact
program outcomes, and whether families will benefit from receiving fewer services than
planned.
The Healthy Families Pilot Program data system provided information about discrepancies
between the program model and actual implementation. DES provided site level enrollment
data that was used to estimate participants' attrition rates. In addition, data pertaining to
participant involvement was gathered during 2 focus groups conducted with Healthy Families
Pilot Program staff during May 1996. Nineteen family assessment workers participated in the
first group. The second group included 16 family support specialists representing sites across
the State.
Enrolled Families Do Not
Receive Program's Full Services
Most families enrolled in the Healthy Families Pilot Program fail to realize all the benefits
offered by the Program. Data show that, in the Arizona program, few families are receiving
the number of home visits outlined by the Healthy Families Pilot Program and almost half of
the families are dropping out before completing it Further, workers report that many families
are not using services to which they are referred.
Families miss scheduled home visits- Most families in the Program are receiving fewer home
visits than planned. The Healthy Families Pilot Program outlines an ideal number of home
visits a family should receive in order to build a caring, trusting relationship with program
workers. The program guidelines state that new families should receive their first home visit
within 3 to 5 days of enrollment. Weekly home visits should occur for at least the first 9
months of a family's participation in the Program. If d m g this period progress is observed,
families move to a new service level, and receive only 2 visits per month. Ideally, a family
should receive 15 to 24 visits during their first 6 months in the Program. Arizona Healthy
Families Pilot Program participants received only 13 home visits, on average, during their first
6 months in the Program. However, families may also participate in group activities, such as
family nights and play groups. These group activities may count as one home visit per month.
Although the Healthy Families Pilot Program is not providing the ideal number of home
visits, this does not appear to be unusual. For example, Tennessee reporb that during the first
5 months of service, its clients received an average of 8 home visits.
Fewer home visits than the service goal can be partially explained by the high number of
attempted but unsuccessful home visits. Family support specialists report they often arrive at
prescheduled home visits only to find no one home. In their first 6 months in the Program,
only 11 percent of the families were home for each scheduled visit and almost 10 percent of
the families did not keep 9 or more scheduled visits. On average, families did not keep 4 visits.
Many factors contribute to families missing home visits. For example, family support
specialists stated that some families do not have phones and consequently cannot call to cancel
appointments. In addition, some families lack personal transportation and miss scheduled
visits when they get a ride to do grocery shopping or other errands. A missed visit can result
in additional weeks missed because of difficulties in reaclung families to schedule the next
session. Furthermore, families often move between scheduled visits without contacting their
Healthy Families Pilot Program support specialist Frequent moves make it difficult to
schedule and attend weekly home visits. Finally, some participants miss their appointments
because they are told by other family members it is not important to participate in the
Program.
To reduce the amount of unproductive time family support specialists spend on missed home
visits and rescheduling these visits, they should schedule each family's visits for the same day
and time for the first six months they are in the Program.
Families drop out of Program before completing services- Despite the importance that the
Arizona Healthy Famllies Pilot Program places on families remaining in the Program, a large
number of families do not complete it The Arizona Healthy Families Pilot Program is
designed to serve a family until the child's fifth birthday. This is to ensure that important
services are delivered during a child's critical developmental period. In addition, studies
indicate a majority of all severe abuse occurs when children are between birth and age five.
The Arizona Healthy Families Pilot Program has an attrition rate of 40 percent for fiscal year
1996, and a 47 percent overall attrition rate since the Program began. The rate ranged from a
low of 39 percent to a high of 49 percent when analyzed by the contractors serving the
families. The attrition rate for the Program does not seem to be extremely high when
compared to other Healthy Family programs and given the high mobility rate of Arizona's
population. Healthy Families programs across the nation have attrition rates that range from
66 percent to less than 10 percent.
Most families who are active in the Program and leave do so because they move. More than
60 percent of the participants who had left the Program as of April 1996 did so because the
family support specialist was no longer able to contact the family or because the family
moved. In addition, almost one- fourth left the Program because they either refused services,
or refused to accept a new home visitor after their original home visitor left the Program. The
remaining families left for a variety of other reasons.
During the next year, efforts will be undertaken to more fully explain the causes of the
Healthy Families Pilot Program's attrition rate, and this information will be presented in the
final report.
Families do not obtain other needed smices- In addition to the difficulties encountered in
visiting families and the fact that families drop out of the Program, participants do not always
take advantage of other available services. Family support specialists report that a lack of
understanding about the importance of preventive health care, medical care, immunizations,
and proper nutrition mates barriers to clients seeking the services they need and are referred
to. For example, some families assume that seeing a doctor is done only when one is ill.
Additionally, some families believe childhood immunizations are not needed until a child
begins school.
Furthermore, many families have no way to get to needed services such as medical
appointments, Women, Infants and Children feeding programs, immunization clinics, and
other services. Wlule many Healthy Families Pilot Program support specialists will provide
transportation when families have no alternatives, they often incur risks, since their employer
may not provide their insurance for transportation or reimbursement for mileage. Providing
transportation also limits the time that workers can serve other clients.
Families That Need Services
May Not Enroll or Actively
Participate in the Program
Although the Healthy Families Pilot Program has successfully used a variety of methods to
enroll participants, many families still fail to enroll or actively participate. The Program's
enrollment goals have been exceeded; however, screening procedures limit some eligible
families from enrolling. In addition, families may enroll in the Program, but never become
active participants.
Families that need smices may not enroll in the Program- The Healthy Families Pilot
Program uses a variety of methods to enroll participants, but does not always enroll families
who need its services. The program's policy allows for family enrollment, usually shortly after
a baby's birth, using four different procedures. The methods vary according to the roles of
hospital personnel and the Program's assessment workers in screening and enrollment, and
whether the screening is done in the hospital or the family's home. Not all sites use all the
procedures, because some have more staff than others. The Program exceeded its targeted
enrollment of 887 participants for fiscal year 1995 and fiscal year 1996. As of June 30,1996, the
Program has enrolled 1,329 families. However, in fiscal year 1996, 285 families refused
assessment and 216 families who met the risk criteria did not enroll in the Program. As a
result, some families needing services were not enrolled in the Program.
Family assessment workers report several causes for not enrolling families. First, these
workers face a time constraint when attempting to enroll families into the Program. Workers
attributed this time constraint to quick hospital discharges and the fact that there must be one
hour between screening clients for the Program and describing the Program, and offering it
to the family. During this one- hour wait, mothers and babies may be discharged from the
hospital, requiring the family assessment worker to conduct the second screen at a later date
in the family's home.
Second, family assessment workers often work part- time, and are often not available to screen
those mothers who give birth and are discharged during the weekend. This reduces the
number of clients a family assessment worker can enroll. In addition to time constraintsI
family assessment workers also report that the likelihood of families enrolling in the Program
is reduced by the influence of other family members. Specifically, the presence of family
members may lead mothers to not answer questions truthfully on program enrollment forms.
Additionally, family members may prompt mothers to decline the Program. Finally, some
families may not be enrolled because the Program is at full capacity at the time of assessment.
To address these issues, programs could rely more on home- based screening procedures.
Additionally, programs can screen weekend births and discharges by using on- call family
assessment workers.
Many families enrolled in the Program are not active participants- A high number of
program participants enroll in the Program, but never become active participants. After
enrollment, staff have a one- month period to complete a successful home visit with the family.
If a home visit cannot be completed within a month, families are placed on " creative
outreach," whch includes phone calls, numerous attempted home visits, and meetings at
other locations to get the family actively enrolled in the Program. If after two months of
creative outreach the family is not receiving ongoing services, they are dropped from the
Program. Of 257 famihes who were in creative outreach, three- fourths have been dropped as
of April 1996. The consequence is that almost 40 percent of the families dropped from the
Program as of April 1996 were enrolled, but were never active in the Program. Establishing
a date and time for the first home visit at the time of enrollment could reduce the number of
families who are placed on creative outreach.
Limited Family Participation
Impacts Program Outcome
Major obstacles to service delivery include families not being home, high attrition, and
families' lack of knowledge and understanding about services offered. Since the Program
provides services to families who often have no positive role models and who lack the
financial means for stable housing, transportation, and telephones, these obstacles to program
delivery are not surprising. These barriers result in considerable deviation from the program
model in terms of families' length of enrollment in the Program ( duration) and frequency of
visits ( intensity).
However, it is not yet clear how deviation from the program model in terms of duration and
intensity will affect program outcomes and impact, and if families who receive limited services
benefit from them.
Recommendations
1. Program sites should use all of the established procedures to intake families into the
Program to increase the number of eligible families who are enrolled.
2. To ensure families are home for visits:
a. the first home visit date should be established at the time of assessment, and
b. each family support specialist should have established days and times for each family's
home visits.
( This Page Intentionally Left Blank)
Other Pertinent Information
During the course of the evaluation we obtained other pertinent information about the
Healthy Families Program.
National Concerns About the
Family Stress Checklist
There is concern that the Family Stress Checklist may screen too many families into the
Program. While there is currently no data available to Mly explore th~ psr oblem, efforts are
being made by the Healthy Families America Research Network ( Network) to address this
situation? The Network is investigating ways to improve and refine the checklist to increase
its ability to better measure a familfs risk level, Because of the way this instrument is
constructed, simply raising the eligibility score will not solve the problem. The Networlk's
recommendations should improve the instrumenfs reliability and validity.
1 Healthy Families America Research Network is supported by a grant from the Camegie Corporation. This
group includes evaluators from 28 Healthy Families programs across the country, and meets twice a year to
address evaluation of policy issues.
( This Page Intentionally Left Blank)
STATUTORYANNUAL
EVALUATION COMPONENTS
Pursuant to Laws 1994, Ninth S. S., Ch. 1, § 9, the Office of the Auditor General is required to
include the following information in the annual program evaluation.
1. Information on the number and characteristics of the program participants.
Information on the number and characteristics of program participants was available only
through April 30,1996. Additionally, demographic information varies in completeness and
reliability. This report contains demographic information only on those participants for
whom data were available. Because mothers are more involved in the Program than
fathers, more complete information was available for them than for fathers. For both
mothers and fathers there is variation in the valid information from item to item. For
example, we have more information about participants' ethnicity than their education. We
have demographic data on 1,045 families who were enrolled in the Healthy Families
Program as of April 30,1996. This data includes current and disenrolled families.
Participants by County- At the end of April 1996, the Healthy Families Pilot Program
was serving 699 families. Approximately 57 percent of these families reside in the
urban areas of Maricopa and Pima Counties. The remaining families are located in four
predominantly rural counties - Coconino, Yavapai, Cochise, and Santa Cruz.
Participants by Age, Employment, and Education- The median age of mothers
participating in the program is 22. Additionally, 27 percent of Healthy Families Pilot
Program mothers are teenagers. While 39 percent of mothers report they have either
a high school diploma or GED, 89 percent of mothers report being unemployed.
Since fathers are often not involved with the baby or mother at the time of birth,
getting their demographic information is difficult. Given this, there are 617 reported
cases with information on the fathefs education. Of these 617, 45 percent have
completed the 12th grade. Sixty- four percent of fathers report being employed, and
their median age is 25.
Other Demographic Information - Additional demographic information on Healthy
Families Pilot Program participants includes participants' marital status, living
situation, ethnicity, and household income. Eighty- three percent of mothers reported
being single, separated, or divorced. In addition, 32 percent of the mothers lived with
their parents. Eighteen percent of mothers in the Program live with a spouse and 24
percent of parents live with a non- relative. Approximately half of the Healthy Families
Pilot Program mothers ( 52 percent) have one child.
There is also variation in the participants' ethnicity. Forty- nine percent of the mothers
and 52 percent of the fathers are Hispanic. Non- Hispanic whites account for 29 percent
of the mothers and 24 percent of the fathers. Approximately 12 percent of the mothers
and 12 percent of the fathers are Native American.
Most program participants belonged to impoverished households. Among 831 family
households whose yearly household income was reported ( not including food skimps),
almost three- quarters ( 72 percent) reported annual household income below $ 10,000.
The median income is $ 6,000 per family. Only 6 percent reported an income above
$ 20,000. A large number of families depended on one or more welfare benefits, most
commonly Aid to Families with Dependent Children, food stamps, and Women,
Infant, and Children programs. Additionally, only 10 percent of families reported
having private medical insurance, wMe over 80 percent were enrolled in the Arizona
Health Care Cost Containment System.
Many of the mothers in the Program also have had personal problems that can make
the challenges of raising their new babies more difficult For example, 30 percent of
mothers in the Program have a history of substance abuse, 25 percent of mothers
reported that they had a history of depression, and 15 percent have a history of
psychiatric care. However, fewer than 1 percent of the babies tested positive for alcohol
at birth and fewer than 2 percent tested positive for drugs.
Information on contractors and program service providers.
DES awarded 5 contracts to serve 13 sites in 6 counties. Tucson Association for Child Care
( Tucson) received two separate contracts for urban and rural sites, and manages three
urban and three rural not- for- profit service sites. In addition, its urban contract includes
the cost of providing statewide program oversight through a quality assurance
coordinator. Southwest Human Development ( Southwest) was awarded a contract for
managing four sites in Maricopa County. The fourth contract was awarded to the Marcus
J. Lawrence Medical Center ( Lawrence), which operates one site in Yavapai County. The
h a 1 contrador, Coconino County Department of Public Health ( Coconino), has two sites,
both serving Coconino County.
In addition to these site contracts, DES contracted out the database management function
to a data management firm.
3. Information on program revenues and expenditures.
Three million dollars was appropriated to the Healthy Families Pilot Program for the year
ended June 30, 1996. The Program received another $ 65,000 from federal sources and
$ 442,759 of support from the contractors. These revenues combined to a total of $ 3,507,759.
Of these revenues, $ 3,256,112 was spent on the Program and $ 251,647 of unspent funds
was carried forward by contractors to support 1997 activities. Table 1 details fiscal year
1996 revenues and expenditures by contractor.
Healthy Families Pilot Program
Revenues and Expenditures by Contractor
Year Ended June 30,1996
( Unaudited)
Tucson Tucson Tucson Data
Urban Rural Oversight Southwest Lawrence Coconino Management Total
Revenues
State and federal $ 661,908 $ 666,908 $ 182,864 $ 1,007,577 $ 140,127 $ 328,2288 $ 77,328 $ 3,065,000
Contractor contributions 101,642 75,410 2,000 189,000 39,130 35577 442,759
Total revenues 763250 742,318 184,864 1,196377 179,257 363m 77,328 3,507,759
Balance, June3 0,1996 $ 75,000 $ 54,258 $ 0 $ 64577 $ 32591 $ 25,221 $- -
Source: Auditor General staff analysis of financial information provided by the Department of
Economic Security.
DES does not use any appropriated monies to administer the Healthy Families Pilot
Program. DES has two FTE to administer the program and provide technical assistance
to the contractors. One FTE is supported by Child Abuse Prevention Fund monies and the
second is supported by DES, Division of Children, Youth and Families, Administration
for Children, Youth and Families? Figure 4 ( see page 24) shows the distribution of state-appropriated
dollars in fiscal year 1995- 96.
1 The Legislature established the Child Abuse Prevention Fund in 1992 without appropriated dollars, to promote
child abuse prevention and provide financial assistance to community- based agencies for this purpose. The
Child Abuse Prevention Fund is supported through a voluntary check- off on state income tax returns.
4. Information on the number and characteristics of enrollments and disenrollment.
Enrollment and disenrollment information was obtained from a different source than
program participants' characteristic data and has a different cutoff date. Enrollment
and disenrollment data contains data from January 1995 through June 1996. As of
June 30,1996, the Program has enrolled 1,329 families since it first began to provide
services in January of 1995. During that same period, 629 families have left the
Program. This is a 47 percent attrition rate. Fiscal year 1996 began with 519 families
enrolled from fiscal year 1995. An additional 721 families were enrolled in fiscal year
1996, and 537 families left the Program during this same period. Thus, for fiscal year
1996, there is a 40 percent attrition rate.
Figure 4
Healthy Families Pilot Program
Distribution of Expenditures
Year Ended June 30, 1996
( Unaudited)
E3Space OErnployee- Related Expenditures
Olndirect Costs = Material and Supplies
l3Professional and Outside Services mother Operating Costs
Source: Auditor General staff analysis of data provided by the Department of Economic
Security.
During fiscal year 1996, 3,498 families completed a Healthy Families Pilot Program
Hospital Chart Screen through the DES program. Of these families 1,027 were
assessed with the Family Stress Checklist and 937, or 91 percent, tested positive. All
but 213 of these families who tested positive accepted enrollment in the Program.
Families still enrolled have been in the Program an average of 308 days. Families who
have left the Program were enrolled for an average of 168 days. Fewer than 10 percent
of the families who have left the Program did so within the first 30 days.
More than 60 percent of the participants exited the Program for two primary reasons.
First, the most common reason for a family leaving the Program is that the family
support specialist ( home visitor) was no longer able to contact the family. The second
most common reason was the family moving. Almost one- fourth of the families left
the Program because they either refused to continue, or refused to accept a new home
visitor after their original home visitor left the Program.
During the next year we will work with the Program staff to determine the causes for
attrition and procedures that might lower the rate.
5. Average cost for each participant in the Program.
Table 2 ( see page 26) presents the average cost per family, calculated on two different
bases. Method A calculations are based only on families who were active participants
at fiscal year- end. Method B calculations are based on the total number of families
who were served at any time during the fiscal year. Other cost analyses ( such as the
average total cost for a family to complete the Program), requiring additional data
expected to become available during the present fiscal year, will be included in the
next report.
In our 1995 report we predicted that annual costs per participant would decrease as
the Program increased the number of families it serves. Although the Program has
exceeded its enrollment goals, the high attrition rate has caused it to fail to meet and
maintain its service goals. The result is that the second- year costs calculated using
Method A above actually increased. The figures computed for Method B decreased
because the total number of families who have been enrolled, but are not consistently
active in the Program, have increased. If the Programs had met and maintained the
service goal of 917 active families in the Program rather than the 727 families who are
active, the costs would have been $ 3,549, which would have been lower than in 1995.
Table 2
Healthy Families Pilot Program
Cost1 per Family
Years Ended June 30, 199! j2 and 1996
Method A Method B
Based on the number of
families enrolled in the
program at yearend ( does
not include families who
have disenrolled)
Expenditures 1995 1996
State $ 3,269 $ 4,127
Federal and contractor
contributions 424 350
Total cost per family $ 3,693 $ 4,477
Based on the total number
of families served during
the fiscal year ( includes
families who disenrolled)
Costs include quality assurance, training, and data management.
1995 costs are for a six- month service period only.
Source: Auditor General staff calculations are based on data provided by the Department of
Economic Services.
6. Information concerning progress of program participants in achieving goals and
objectives.
This report does not address participants' progress in achieving program goals and
objectives because not enough participants have participated long enough to provide
information on outcome measures. The third and final report will focus on the Program's
effectiveness, and its impact on the program participants.
7. Recommendations regarding program administration.
Overall, the program administration was efficient and the administrative tasks at the
DES level were completed in a timely fashion.
8. Recommendations regarding informational materials distributed through the
programs.
The Healthy Families Pilot Program distributes informational materials in accordance
with the state- mandated services. Our office selected and reviewed materials related
to child development, parent- child attachment, and bonding issues and found them
to adequately address program needs. No recommendation is deemed necessary
regarding informational materials distributed through the Program at this time.
9. Recommendations pertaining to program expansion.
Recommendation regarding program expansion can be made only after the programs
have operated for at least the mandated three- year period and some outcome information
is available.
( This Page Intentionally Left Blank)
Agency Response
I Fife Symington
Governor
Linda J. Blessing, DPA
Director
NOV 1 3
Mr. Douglas R. Norton
Auditor General
Office of the Auditor General
2910 North 44th Street, Suite 410
Phoenix, Arizona 85004
Dear Mr. Norton:
Thank you for the opportunity to review the Second Annual Evaluation of the Healthy
Families Pilot Program.
I am pleased that this report acknowledges the improvements made since the last
evaluation. As you are aware, the ongoing success of the Healthy Families Program
remains central to our prevention efforts.
Additionally, please know that the department will thoroughly analyze and implement
the recommendations contained in this evaluation. In fact, staff have taken the
important first steps to improve the scheduling of home visits and to ensure optimal
program enrollment.
Finally, the department wishes to thank you and your staff for the time and quality
effort devoted to this important evaluation.
Sincerely,
( This Page Intentionally Left Blank)
APPENDIX
Assessment Tools
Hospital Chart Screen
A Hospital Chart Screen is completed at the child's birth to determine if families should
be screened for the Program using the Family Stress Checklist. The screen consists of 15
items which are coded as true, false, or unknown for each potential client. The items measure
a variety of factors that can contribute to child abuse including marital status; late or no
prenatal care; history of substance abuse, abortions, and depression; unsuccessful abortion
or adoption of the baby; and fewer than 12 years of education.
Family Stress Checklist
The Family Stress Checklist, an assessment tool developed at the University of Colorado
Health Services Center, provides an indication of whether or not a family is at risk of
abusing or neglecting their children. The Family Stress Checklist is an unstructured interview
conducted with the families around the time of the child's birth. The Checklist provides
a measure of the family's risk of child abuse and determines eligibility for the Program.
The Family Stress Checklist contains 10 rating factors for which each mother and father
can receive a score of normal ( 0), mild ( 5), or severe ( 10). The ten factor scores are added
to compute a total score. Separate scores are computed for each parent.
Child Abuse Potential Inventory
The Child Abuse Potential Inventory is a self- reporting device that determines the risk
of a parent physically abusing a child. The scale is written on a third- grade level and
includes 160 agree/ disagree items. The physical abuse scale consists of six factors: distress,
rigidity, unhappiness, problems with child and self, problems with family, and problems
from others. The Child Abuse Potential Inventory also includes information about how
consistently and truthfully respondents have answered the questions.
FACES ll
FACES II is designed to measure family dynamics. It measures family dynamics on two
dimensions, cohesion and adaptability. Cohesion is the degree to which family members
are separated or connected to their families. Adaptability is a family's ability to change
a- i
its power structure and role relationships. These two dimensions are combined to give
a general family type score.
HOME
HOME is an observation and interview instrument that measures the quality of stimulation,
support, and structure available to children in their homes. Different forms are used for
three different age groups - 0 to 3 yrs. ( infants and toddlers), 3 to 6 yrs. ( preschoolers),
and 6 to 10 yrs. ( elementary school age). Healthy Families Arizona uses only the form meant
for infants and toddlers. The 0 to 3 yrs. form measures the following six home environment
factors: 1) emotional and verbal responsivity of parent, 2) acceptance of child's behavior, 3)
organization of physical and temporal environment, 4) provision of appropriate play materials,
5) parent involvement with child, and 6) opportunities for variety in daily stimulation.
ASQ ( Ages and Stages Questionnaire)
ASQ is a parent completed, child monitoring system. The questionnaire can be administered
11 times until the child turns 4. However, depending on program needs it can be
administered less. In the case of Healthy Families Arizona, the questionnaire was used
8 times.
The questionnaire addresses the following five areas of child development: l) communication,
2) gross motor, 3) fine motor, 4) problem solving, and 5) personal- social.
a- ii