Annual evaluation, Health Start pilot program

              ANNUAL EVALUATION
HEALTH START PILOT PROGRAM
Report to the Arizona Legislature
By the Auditor General
January 1996
Report 96- 2
STATE OF ARIZONA
DOUGLAS R. NORTON, CPA OFFICE OF THE
AUDITOR GENERAL AUDITOR GENERAL
January 25,1996
11 Members of the Arizona Legislature
DEBRA K. DAVENPORT, CPA
DEPUTY A" 01TOR c. INEI1IL
The Honorable Fife Symington, Governor
Dr. Jack Dillenberg, Director
Arizona Department of Health Services
Transmitted herewith is a report of the Auditor General, an Annual Evaluation of the Health
Start Pilot Program. This report is in response to the provisions of Session Laws 1994,9th
Special Session, Chapter 1, Section 9.
This is the first in a series of three reports. The second and final evaluations are scheduled to
be released on or before December 31, 1996, and December 31, 1997, respectively. Our
evaluation of the Health Start program found that the Arizona Department of Health Services
( ADHS) contract award process should be revised so that it is clear on what basis a
community is selected to receive the Health Start program. We also found that the ADHS
should continue its efforts to coordinate Health Start with other available services. In addition,
we found that as part of the program, the ADHS developed the Arizona Family Resource Guide
which is well designed and informative and was created and distributed at limited cost to the
state.
However, due to problems in program implementation and data collection, we have concerns
at this time about the feasibility of conducting a future outcome evaluation demonstrating
progress in achieving participant goals and objectives.
My staff and I will be pleased to discuss or clanfy items in the report.
This report will be released to the public on January 26,1996.
Sincerely,
Ayu itors GenNe orratlo n
Enclosure
I 2 9 1 0 NORTH 44TH STREET . SUITE 4 1 0 m PHOENIX, ARIZONA 85018 . ( 6 0 2 ) 5 5 3 - 0 3 3 3 rn FAX ( 6 0 2 ) 553- 0051
SUMMARY
The Office of the Auditor General has completed the first in a series of three annual pro-gram
evaluation reports to be issued on the Health Start Pilot Program and the Arizona
Children and Families Resource Directmj, administered by the Arizona Department of Health
Services ( ADHS), Office of Women's and Children's Health ( OWCH). This evaluation
was conducted pursuant to the provisions of Session Laws 1994, gth Special Session, Chap-ter
1, Section 9. The second and final evaluation reports are scheduled to be released on or
before December 31,1996, and December 31,1997, respectively.
Arizona's Health Start Pilot Program is designed to provide children with a healthy start
in life by identifying pregnant women in need of services and providing them with edu-cation,
emotional support, advocacy, and referrals. The Program's goals are to increase
pregnant women's access to prenatal care, reduce the incidence of low birth weight ba-bies,
improve childhood immunization rates, reduce the incidence of children affected by
childhood diseases, provide information about preventive health care, and assist families
in identifying school readiness programs.
The State appropriated $ 975,000 for the Health Start Pilot Program in fiscal year 1994- 95
and $ 1,400,000 in 1995- 96. Other funding sources increased revenues to $ 1,365,584 in 1994-
95 and to $ 1,636,695 in 1995- 96.
The OWCH awarded contracts to 13 Health Start providers to serve participants between
March 1,1995, and June 30,1996. The contractors serve urban communities in Phoenix,
Tucson, and Yuma, and over 60 communities in rural or semi- urban areas across the
State. Providers served 2,740 new and continuing participants and conducted outreach
services to 103 nonparticipants in 1994- 95. The contractors provided services for a total of
$ 287,930 in the first fiscal year and are contracted to provide $ 1,141,240 in direct services
in 1995- 96.
Procedures for Selecting Pilot
Sites Need to Be Improved
( See pages 10 through 13)
Health Start's approach is to target needy communities and serve all pregnant women in
the community who wish to participate. Because Health Start does not have any other
criteria for enrollment, it is important that the communities selected as pilot sites are truly
needy areas in the State. However, ADHS proposal reviewers did not follow its written
criteria when they evaluated the proposals on the factor of demonstrated need for the
Program. To ensure that the basis for community need is documented, the OWCH should
revise the evaluation criteria and methods.
The OWCH should establish evaluation criteria to ensure that funded communities have
a documented need for Health Start. The OWCH should revise its evaluation system by
establishing several criteria under which a community can qualify as having need, allo-cating
more points for documented need in a proposal's overall score, and applying crite-ria
more consistently in evaluating proposals. The OWCH should also consider identify-ing
specific sites with the highest documented need so proposals can be requested to
specifically serve those communities.
Health Start Needs to Continue Its
Efforts to Coordinate with Related Programs
( See pages 14 through 16)
The OWCH needs to increase its efforts to coordinate Health Start with other services
available to pregnant women and their families. Because Health Start's target population
is broadly defined, many other Arizona programs may be serving families with needs
similar to those addressed by Health Start. Without adequate coordination, Health Start
may be serving participants who would be better served by another program. Although
the OWCH has recognized the need for coordination, and is working to develop a system
to coordinate efforts, many barriers to coordination remain. The ADHS should conduct a
formal study to assess the feasibility of comprehensive program coordination.
Health Start Reverts a Significant
Amount of Its Appropriation
( See pages 18 through 19)
Health Start will revert a significant amount of its first- year appropriation to the State
General Fund. At least $ 500,000 of the state funding for Health Start will revert to the
State General Fund. The reversion is caused by a shortened service year and the Program's
reimbursement requirements, under which OWCH does not pay contractors for services
until the participant's baby is born. In the future, the Legislature may want to consider
either providing only planning funding for the first year of a pilot program, or providing
non- reverting funding for the first year.
Arizona Family Resource
Guide Meets Goals
( See pages 20 through 22)
The goal of the Arizona Family Resource Guide is to aid parents in accessing needed services
without relying on public programs for referrals. The Guide is well developed and infor-mative.
It fulfills its goal of increasing parents' access to information, while its size and
format allows it to be widely distributed at limited cost to the State. Before reprinting the
Guide, the OWCH should make a few minor changes to verify information and correct
minor problems.
Statutory Annual
Evaluation Components
( See pages 24 through 29)
Responses to legislative questions regarding program participants, contractors, revenues
and expenditures, enrollment and disenrollment in the Program, the cost per participant,
and recommendations are described in detail in the final section of this report.
This report does not address the progress of program participants in achieving goals and
objectives, because the program was operational for only four months during the first
year. However, the Office of the Auditor General has specific concerns about the feasibil-ity
of demonstrating progress in achieving participant goals and objectives. First, the
Program's goals and expected outcomes are inconsistently defined by program
implementers. Second, the methods used for selecting sites and participants may make it
difficult for the Program to have a demonstrable impact on outcomes. For example, some
communities receiving Health Start have more favorable rates of low birth weight or
prenatal care than the statewide rates. Finally, the Program collects very limited informa-tion
on participants, restricting the scope and comprehensiveness of the outcome evalua-tion.
iii
( This Page Intentionally Left Blank)
Table of Contents
Introduction and Background ........................................................
Finding I: Procedures for Selecting
Pilot Sites Need to Be Improved ...............................................
Background .........................................................................................................
Methods for Selecting Pilot
Sites Need Improvement .................................................................................
Recommendation ...............................................................................................
Finding 11: Health Start Needs to Continue
Its Efforts to Coordinate
with Related Programs ..............................................................
Health Start's Target Population
Is Broadly Defined .............................................................................................
OWCH Efforts to
Increase Coordination ........................................................................................
Recommendation ...............................................................................................
Finding 111: Health Start Reverts a
Significant Amount of Its Appropriation ..................................
Short Contract Period ........................................................................................
Reimbursement Policy .......................................................................................
Alternatives to Reversion ................................................................................
Recommendation ...............................................................................................
Table of Contents ( con8t)
Finding IV: Arizona Family Resource
Guide Meets Goals ..................................... ... .............................
Background ................... .. ............ .. .. ... ...
Paae
20
20
Well- Developed Guide
Meets Its Goals ...............................................................................................
Recommendation ........................... .. ...............................................................
Statutory Annual
Evaluation Components ......................................................
Agency Response
Appendix A .................... ... ......................................................... a- i
Appendix B ......................... ... ...................................................... a- v
Appendix C ............................ ..... ........................................... a- ix
Appendix D ................................................................................... a- xv
Table 1
Table 2
Table 3
Health Start Pilot Program Contracts
for Fiscal Years 1994- 95 and 1995- 96
( Unaudited) .............................................. .. ......................
Points Assigned to
Proposal Scoring Areas ............................ .. .................... a
Health Start Costs
Per Participant Estimates
for Fiscal Year 1994- 95 .................................................... a- xvii
Figure 1
Figure 2
Figures
Health Start Pilot Program Funding
Sources for State Fiscal Year 1995- 96 .............................
Communites Served by Health Start
March 1995 through June 1996 .......................................
vii
Paqe
5
INTRODUCTION AND BACKGROUND
The Office of the Auditor General has completed the first in a series of three annual pro-gram
evaluation reports to be issued on the Health Start Pilot Program and the Arizona
Children and Families Resource Directory, administered by the Arizona Department of Health
Services ( ADHS), Office of Women's and Children's Health ( OWCH). This evaluation
was conducted pursuant to the provisions of Session Laws 1994, gth Special Session, Chap-ter
1, Section 9. The second and final evaluation reports are scheduled to be released on or
before December 31,1996, and December 31,1997, respectively.
History of the Program
Health Start has served communities in Arizona since 1988. Initially, the Program used
lay health workers to conduct community outreach and provide home visits to women
during their pregnancies. In 1993, the Program expanded to emphasize family health
services, and added a follow- up period in which families received home visits until their
child was two years old.
The Arizona Children and Families Stability Act, enacted during the 1994 9th Special
Session ( Laws 1994,9th S. S., Ch. 1 § 8), created a new Health Start Pilot Program. This Act
built upon the previous program, but expanded the length of the family follow- up period
from two to four years after birth.
Health Start Goals
Arizona's Health Start Pilot Program attempts to provide children with a healthy start in
life by identifying pregnant women needing services and providing them with educa-tion,
emotional support, advocacy, and referrals. The Program's specific goals are to in-crease
pregnant women's access to prenatal care, reduce the incidence of low birth weight
babies, improve childhood immunization rates, reduce the incidence of children affected
by childhood diseases, provide information about preventive health care, and assist fami-lies
in identifying school readiness programs.
Timely and adequate prenatal care can reduce the incidence of low biqth weight, which in
turn can improve the child's health, reduce health care costs, and reduce infant mortality.
Although it is important for pregnant women to receive prenatal care in the first trimes-ter,
only 69 percent of Arizona's pregnant women receive such care. A recent study by the
Arizona State University School of Health Administration and Policy found that in Ari-
zona, teenagers, high school dropouts, single women, and women of ethnic minority in
particular need more education about the importance of early and adequate prenatal care.
The rate of low birth weight babies born in Arizona has risen, from 6.1 percent in 1983 to
6.7 percent in 1993.
One of Health Start's primary goals is to assist women in accessing prenatal care. A com-mittee
from the National Academy of Sciences' Institute of Medicine has found that, even
when financial and institutional barriers to care are reduced, there are many women who
will still not receive prenatal care. They may experience sociodemographic or attitudinal
barriers that prevent them from obtaining needed care. For example, even if the process
for obtaining Medicaid coverage were simplified, some women would not receive prena-tal
care because they do not understand English or because they are afraid of doctors. The
Committee found that certain types of outreach and social support programs ( such as
those providing services like Health Start's) can be successful at reducing sociodemo-graphic
and attitudinal barriers and helping pregnant women obtain the needed prenatal
services.
Program Description
Health Start serves pregnant women, their children, and their families. A woman is eli-gible
to enroll in the pilot program if she is pregnant and resides within a contractor's
service area. Although there are no risk- based eligibility criteria for participants, the Pro-gram
is designed to serve communities in which much of the population is at risk of poor
maternal health care, low birth weight, low rate of immunization, or generally poor early
childhood health.
The Health Start Pilot Program is based on a lay health worker ( often called promotura)
model for service delivery. In the promotura model, lay individuals are hired to provide
outreach and to promote health care behaviors in their communities. Lay health workers
are typically trusted members of their communities, who reflect the ethnic, cultural, and
socioeconomic characteristics of the residents. Because they are nonprofessional workers,
they are not required to have prior training or experience in health care. The lay health
worker does not provide any direct health services, but does encourage members of the
community to access health services as appropriate. They also act as advocates and role
models for community members.
In the Health Start Pilot Program, lay health workers provide community outreach, pre-natal
home visits, and four years of follow- up home visits after birth (" family follow- up")
to women in populations that are at risk. The main purposes of prenatal home visits are to
ensure that the participant obtains prenatal medical care from a physician and to educate
her about prenatal health issues. The lay health worker also provides support, referrals,
and assistance on other topics as appropriate, including assistance in accessing financial
aid, in coping and problem- solving, and in finding other resources families may need.
Family follow- up visits focus on promoting preventive child health care ( including im-munizations
and establishing a medical home for each family member, such as a clinic or
HMO), providing basic perinatal and child development education, and providing nec-essary
referrals ( including referrals to early childhood education programs). Although
the family follow- up visits focus on the target child ( the pregnancy during which the
participant was enrolled), the lay health worker promotes health and provides referrals
for the whole family. ( See Appendix A, page a- iii for a list of program activities and
anticipated outcomes for the outreach, prenatal, and family follow- up periods.)
Administration, Budget,
and Administrative Responsibilities
Responsibility for Health Start and the Arizona Children and Families Resource Directory was
assigned to the Arizona Department of Health Services ( ADHS), Office of Women's and
Children's Health ( OWCH), in the Bureau of Community and Family Health Services.
The OWCH oversees 19 programs and 8 projects that relate to issues such as access to
primary care, maternal health, child health, and injury prevention.
Healtlz Start revenues - The State appropriated $ 975,000 for the Health Start Pilot Pro-gram
in fiscal year 1994- 95 and $ 1,400,000 in 1995- 96. The OWCH has used the state pilot
program appropriation and funding from other sources to develop and administer a single
Health Start program. Other revenue sources include an existing state appropriation for
prenatal outreach, funds from federal block grants for Maternal and Child Health ( MCH),
and private grants from the National Association for the Education of Young Children
( NAEYC). The 1995- 96 revenues from each source, totaling $ 1,636,695, are shown in Fig-ure
1, page 5. ( This figure includes $ 1.6 million in state dollars plus a $ 36,695 allocation
from the 1995- 96 MCH block grant; however, the final allocation from the MCH block
grant may change over the course of the year, supplementing revenues with up to $ 200,000
in federal funds).
Staffing and adltzinistrative responsibilities - The OWCH will utilize 4.0 FTE to admin-ister
the pilot program in 1995- 96 ( 3 FTE will be supported by Health Start funds and the
other FTE will be supported by the OWCH). These staff include the program manager, a
field coordinator, a secretary, a data entry clerk ( half- time) and a systems analyst ( half-time).
The OWCH will also cover the cost of any computer support that is needed from
the ADHS Information and Technology Services.
In 1994- 95, the OWCH developed a number of materials in an attempt to deliver a quality
program that is suitably standardized across sites. These changes were made in response
( This Page Intentionally Left Blank)
Figure I
Health Start Pilot Program Funding Sources
for State Fiscal Year 1995- 96
Arizona Children and Families
Stability Act ( State)
$ 1,400,000
,\
A
Maternal and Child
Health ( MCH) Block
Grants ( Federal)
$ 36,695 a
' Prenatal Outreach
a These carry- over funds may be supplemented by up to $ 200,000 from the 1995- 96 block grant.
Source: Udorm Statewide Accounting System data supplied by ADHS Business and Financial Ser-vices.
to new legislative mandates and to follow up on recommendations from previous Health
Start Program evaluations. Program developments include the Policy and Procedure
Manual, a training curriculum for lay health workers, informed consent forms and bro-chures,
standardized data collection forms, and a contractor on- site review guide. In
addition, the OWCH developed a participant database, which is being used to track all
new and continuing participants.
Contractors and pilot sites - The OWCH awarded 13 contractsfor services.' The first 11
contracts were funded based on a Request for Proposal ( RFP) process. Two additional
contracts were funded subsequent to the RFP process, when it was determined that funds
allocated for service remained available. ( See Table 1, page 6 for a list of contractors.) The
contractors serve urban communities in Phoenix, Tucson, Yuma, and over 60 communi-
One contractor ( a private provider) canceled their contract during 1994- 95, leaving 12 current con-tractors.
Table 1
Health Start Pilot Program Contracts for
Fiscal Years 1994- 95 and 1995- 96
( Unaudited)
Services
Approved for Contract
Pavment Award
Contractor Service Area ( 1 964- 951b ( 1 995- 96)
I County Health Departments ( 3- 4 months) ( 9- 12 months)'
Cochise County Health and
Social services Douglas and Bisbee $ 24,000 $ 50,580
Coconino County Dept. of
Public Health" Page and other areas 0 66,400
Pima County Health Dept. Tucson and rural areas 19,110 47,000
Pinal County Dept. of
Public Health El0 y 32,580 88,620
Yavapai County Health Dept. Various communities 21,120 70,500
Yuma County Dept.
of Public Health Yuma and other communities 43,740 155,700
I Area Health tducation Centers 1
Northern Arizona Hopi, Navajo, other Reservation Areas
Health Education Center ~ avaj; County ( 1994- 95/ 1995- 96);
Communities in La Paz and
Mohave Counties ( 1995- 96 only) 27,290 227,200
Western Arizona Area Communities in La Paz and
Health Education Center Mohave Counties 13,160 0
[ Community Health CenterslBehavioral Health Centers 1
Centro de Amistad, Inc. Guadalupe 15,000 72,900
Clinica Adelante Migrant Leas around Phoenix 21,850 95,240
Indian Community Health Native Americans in metropolitan
Service, Inc. Phoenix 18,250 47,000
Mariposa Community Nogales and Rio Rico 35,940 127,200
Mountain Park Health Center, Inc. South Phoenix 15,890 92,900
Total * $ 287,930 $ 1,141,240
Source: Auditor General staff analysis of proposals and contracts for Health Start, OWCH summary
map of Health Start contractors and sites, and participant database Health Start.
" This contractor will provide services through a subcontract to Northern Arizona Health Outreach, a
private provider.
These figures are estimated service expenditures for the new ilot program during the state fiscal
year 1994- 95. These figures are based on participants approved for payment according to the OWCH
participant database as of November 21,1995.
Figure 2
Communities Served by Health Start
March 1995 through June 1996
So
Source: OWCH- Health Stari Field Coordinator.
ties in rural or semi- urban areas across the State ( see Figure 2, page 7). Eight providers
have experience in administering the previous Health Start program or similar programs
funded through other sources. See Finding I, pages 10 through 13, for a discussion of the
methods used in selecting pilot sites and the distribution of sites in Arizona.
A total of 2,843 women were identified as " active" in the Health Start database during the
four- month service period of March 1, 1995, through June 30, 1995. Of these potential
participants, 2,740 were enrolled in Health Start and 103 received outreach services but
did not enroll in the Program.
Evaluation Scope,
Responsibilities, and Methods
As mandated by the Legislature, the Office of the Auditor General will conduct annual
program evaluations of the Health Start Pilot Program to examine the effectiveness of the
Program, its organizational structure and efficiency, the type and level of criteria used to
establish eligibility, and the number and demographic characteristics of persons who
receive services from the Program. Two additional program evaluation reports are sched-uled
to be released in December 1996 and December 1997.
This first- year report describes financial information and participants enrolled during
the first pilot year ( state fiscal year 1994- 95).' It also describes program administration
and other activities that occurred during the first pilot year and at the beginning of the
next fiscal year, through December 1995. The report includes activities supported by all
revenue sources for the Health Start Pilot Program, and includes the following findings:
The need to improve the pilot site selection process to ensure it is clear on what basis
a community is selected to receive program services
The need for Health Start to continue its coordination with related programs
The reversion of half of Health Start's 1994- 95 appropriation to the State General Fund.
Methods used in preparing the Arizona Family Resource Guide.
Expenditure information regarding services to contractors was calculated based on services deliv-ered
to participants and approved for payment according to the OWCH participant database, as of
November 21, 1995. All other financial information was calculated from data supplied by ADHS
Business and Financial Services, reflecting 1994- 95 budgets and expenditures current on the Uniform
Statewide Accounting System as of September 30,1995.
The report also contains responses to each evaluation question posed in Session Laws
1994, gth Special Session, Chapter 1, Section 9.
It is too early in the Program to evaluate participants' progress in achieving program
goals and objectives. We expect the 1996 and 1997 reports to address participants' progress
in achieving program goals and objectives, the Program's effectiveness, the long- term
savings of the Program, and other issues.
This evaluation used the following methods: interviews with agency and program staff,
analyses of program revenues and expenditures; analyses of participant enrollment and
characteristics from the participant database; reviews of the Policy and Procedure Manual
and other program materials; content analyses of program proposals, scoring criteria,
and readers' scores; interviews with Health Start contractors and staff; direct observation
of two pilot sites; interviews with coordinators from related programs, reviews of their
program materials; interviews with contractors hired to conduct activities related to Health
Start; reviews of various materials created by program coordination committees; litera-ture
reviews; and discussions with staff at resource agencies listed in the Arizona Fanzily
Resource Guide.
FINDING I
PROCEDURES FOR SELECTING PILOT SITES
NEED TO BE IMPROVED
The OWCH's method for selecting pilot sites should be improved to ensure that it is clear
on what basis a community receives Health Start. The OWCH should revise its proposal
evaluation criteria and methods to more consistently and appropriately evaluate contrac-tor
proposals. Also, in any future procurements for Health Start services, the OWCH
should consider targeting the neediest communities and finding contractors to serve them,
rather than contractors specifying service areas.
Background
Health Start is a community- based program that serves pregnant women, their children,
and their families. Unlike many social programs, Health Start does not screen or assess
participants to ensure it serves individuals with true need. Instead, the program identi-fies
communities in need, and serves members of that community. Session Laws 1994, gth
Special Session, Chapter 1, Section 9 recognize Health Start as a community program and
specify that the evaluation of pilot proposals would include, at a minimum, the following
criteria for demonstrating community need: a high incidence of inadequate prenatal care,
inadequate infant health care, low birth weight babies, or inadequate early childhood
immunizations.
The OWCH used two primary indicators of community need: ( 1) the rate of low birth
weight and ( 2) the rate of inadequate prenatal care ( defined as 0 to 4 prenatal visits).
These indicators were combined to set specific criteria a proposal must meet to receive a
N maximum," " moderate," or " minimum" need score. To show at least moderate need,
the proposal had to demonstrate that the targeted neighborhood or community exceeds
state or county averages for low birth weight and inadequate prenatal care. The proposal
must also demonstrate need in the areas of infant health care and early childhood immu-nizations
to receive a maximum need score.
The OWCH funded all 11 proposals received in response to Health Start's Request for
Proposals ( RFP). ( This does not include a 12th proposal that was later withdrawn by the
contractor.) An additional two contractors were funded subsequent to the RFP process,
using alternative procurement methods. These two contractors have not been evaluated
on the proposal scoring criteria. Each contractor serves from 1 to 23 Arizona communi-ties.
Methods for Selecting
Pilot Sites Need Improvement
The process for selecting pilot sites should be improved to ensure that it is clear on what
basis a community receives Health Start. Under the current process, the criteria were
stricter than necessary, and reviewers failed to follow the evaluation criteria and incorpo-rated
other factors to assign need scores. The OWCH should revise its evaluation criteria
and methods to better target communities needing Health Start.
Evaluation miteria were stricter tlmn necessanj and were not followed - The evaluation
criteria OWCH established to assess relative need required a community to demonstrate
higher than average statistics for both low birth weight and inadequate prenatal care.
However, these criteria appear to have been too restrictive, as some of the Program's
target populations tend to have higher birth weight babies. While these populations may
be unable to document need as measured by rates of low birth weight, they may have
other needs for Health Start, including assistance in accessing prenatal care and educa-tion
about child development and preventive health care.
Proposal reviewers failed to follow the evaluation criteria and assigned higher need scores
than were allowed by the criteria. Auditor General staff reviewed the proposals and evalu-ated
documented need strictly according to the preset criteria. None of the proposals met
the criteria to receive at least a moderate need score, yet most of them received scores in
the moderate to maximum range. Reviewers assigned scores in the moderate to maxi-mum
range even when no statistics were provided for some of the target communities,
and even when communities demonstrated better than average birth statistics.'
Program staff report that other, nonspecified criteria were also used to determine need,
but that these criteria were not formalized or documented. Reviewers seemed to incorpo-rate
other community problems and needs listed in the proposals, such as high teen preg-nancy
rates, a general lack of medical services in the community, and a high rate of pov-erty.
They also seemed to incorporate other definitions of inadequaie prenatal care, in-cluding
the rate of entry into prenatal care after the first trimester.
For some of the proposals missing data on inadequate prenatal care, this omission may have resulted
from lack of specificity in the RFP. Although the evaluation criteria defined inadequate prenatal care
explicitly ( zero to four prenatal visits), the specific evaluation criteria were not contained in the RFP,
so proposal writers had no way of knowing what statistics were required for this section. Four pro-posal
writers reported a different ( although commonly used) statistic for inadequate prenatal care,
starting prenatal care after the first trimester of pregnancy. Future RFPs should specifically define
, I inadequate prenatal care" so that proposed contractors know what statistics are required for pro-posal
evaluations.
OWCH should revise evaluation & teria and methods - In future procurements, the
OWCH should revise its evaluation criteria and methods by which it assigns scores to
need. Rather than requiring a community to demonstrate higher than average scores for
both low birth weight and inadequate prenatal care, the OWCH should allow communi-ties
to qualify as having high need on any of several variables. Community need should
be weighed more heavily in the proposal's overall score and the OWCH should consider
specifying needy sites, rather than allowing contractors to identify target communities.
Evaluation criteria should identify the need upon which a community receives Health
Start. In addition, criteria should seek to target communities that have a specific need for
the Health Start program, not just a high need for health care in general. Communities
with high teen pregnancy rates probably have a higher need for a prenatal program that
focuses on teenagers, such as Teen Prenatal Express, than for a program like Health Start.
Although such additional factors can be included, their importance should be evaluated
in light of the program's model, goals, and specific target population. The criteria should
also demonstrate that the communitfs needs are not met by existing resources. Estab-lished
criteria then need to be documented and applied consistently across the State. With
evaluation criteria that better represent community needs, reviewers will be less likely to
use other criteria in assigning scores.
The criteria developed to evalu-ate
competing pilot program
proposals should put more em-phasis
on a community's needs
than is currently given. A com-munity's
need now accounts for
too few points out of the total
score ( only 5 out of a possible
31 points). Using this strategy,
a proposal demonstrating no
need could have received fund-ing.
In any future procurement
for Health Start services, the
OWCH should make commu-nity
need account for more
points out of the total possible
points a proposal can receive, or
should establish cutoff scores
on the need criteria. Table 2
shows the current point values
assigned to different scoring
areas.
In the current site selection process, the OWCH allowed contractors to specify the com-munities
they intended to serve. Instead, the OWCH should consider targeting specific
sites with the highest need and distributing RFPs specifically to serve them. Some of the
neediest communities might never receive services under the current strategy, because
they have few providers in the area and are thus the least prepared to respond to an RFP.
For example, some communities in Apache County have higher than average rates of low
birth rates and of inadequate prenatal care, but did not submit proposals to receive Health
Start. In such communities, the OWCH may need to provide technical assistance and/ or
development grants to providers in order to prepare them to respond to an RFP.
RECOMMENDATION
The OWCH should set criteria to ensure that funded communities have a documented
need for Health Start. The OWCH should revise its evaluation criteria and methods in the
following ways:
1. Establish several variables under which a community can qualify as having high need.
2. Allocate more points for community need in a proposal's overall score,
3. Apply criteria more consistently in evaluating proposals.
4. Consider identifying specific sites with the highest need and requesting proposals
specifically to serve them.
FINDING I1
HEALTH START NEEDS TO CONTINUE
ITS EFFORTS TO COORDINATE
WITH RELATED PROGRAMS
The OWCH needs to increase its efforts to coordinate Health Start with other services
available to pregnant women and their families. Because the Program's target population
is broadly defined, other programs may be serving families with needs similar to those
served by Health Start. The OWCH has recognized the need, and has begun to develop
coordination between Health Start and other programs. However, many barriers to coor-dination
remain.
Health Start's Target Population
Is Broadly Defined
Because Health Start's target population is broadly defined, many other Arizona pro-grams
may be serving families with needs similar to those served by Health Start. Health
Start has no participant eligibility criteria, and as a result, it may be enrolling participants
who would be better served by one of these other programs.
Arizona has many programs that target pregnant women and their babies and families.
These Arizona programs target at least some goals that are similar to Health Start's goals
or deliver similar services to women, children, and families. Auditor General staff identi-fied
at least 13 programs that appear to parallel Health Start's prenatal goals or provide
similar prenatal services, and at least 20 programs that appear to match some of Health
Start's postnatal goals or provide similar family follow- up services. ( See Appendix B,
page a- vii, for a listing of these programs.) Following are some of the programs most
closely related to Health Start:
Teen Prenatal Express - Teen Prenatal Express, administered by the OWCH, pro-vides
case management and support to pregnant teens ( 17 years old or younger) dur-ing
the prenatal period, and for a limited time after birth. Home visits are provided by
a nurse case manager or social worker.
First Steps - The First Steps program, overseen by the Arizona Chapter of the Na-tional
Committee to Prevent Child Abuse, provides education, referral, advocacy and
support to women with new babies. The program provides either three months of
telephone calls or one year of home visits, depending on the location of the program.
Calls and visits are provided by trained volunteers.
14
Healthy Families - The Healthy Families Pilot Program, administered by the Be-partment
of Economic Security, provides family support and referrals to families with
high family stress who are at risk for child abuse. The program also provides educa-tion
about health goals such as immunizations and preventive health care. Home vis-its
are provided from birth until the child is five years old, by a trained nonhealth
professional.
Baby Arizona - Baby Arizona, administered through the Arizona Health Care Cost
Containment System ( AHCCCS), strives to improve the use of prenatal care in Ari-zona
through service coordination and increased public awareness. Baby Arizona re-duces
barriers to obtaining prenatal care through AHCCCS, so that women can apply
for AHCCCS at a physician's office on their first prenatal visit.
In addition, pilot sites may be overserving women who do not need Health Start's assis-tance
to access prenatal health care. At least one site recruits many Health Start partici-pants
through a local health clinic where they are already receiving medical services. All
sites may be serving some women who could access prenatal care through the State's
Baby Arizona Program, without the assistance of Health Start.
OWCH Efforts to
Increase Coordination
The OWCH has initiated efforts to more comprehensively coordinate between Health
Start and related programs. Currently, coordination depends on contractors contacting
and collaborating with the other programs located in their areas. Program coordinators at
some sites are either unaware that other programs are located in their area, or have been
unable to work out collaborations. Some programs coordinate through cross- referrals with
related programs. These programs make decisions at the local level and without specific
guidance by the State to determine how to place families into the most suitable program.
Although local control over coordination can be valuable, it also produces statewide in-consistencies,
and may be driven by contractor needs rather than participant or state needs.
A committee representing the OWCH and Healthy Families is developing a system to
coordinate Health Start and several related home visiting programs. This system would
place families into appropriate programs based on an evaluation of their needs at several
points during and after pregnancy. Before her baby is born, a pregnant woman would enter
Teen Prenahl Express or Health Start At the child's birth, the family may continue in Health
Start or transition into another program depending on their needs. Families with healthy
babies might be assigned to Health Start, Healthy Families, First Steps, or Community Health
Nursing as the lead program, depending on the family stress level and child abuse potential.
Families with dealthy babies might be enrolled in the. Newborn Intensive Care Program,
Community Health Nursing, or the Arizona Early Intervention Program as the lead program,
with the possibility of moving back to Health Start or Healthy Families if and when the child
becomes healthy. Although the family is assigned to one lead program, they may receive
services from another program ( e. g., Community Health Nursing), as other needs arise.
While the OWCH has started to address coordination issues, many barriers to coordination
remain. Coordination issues can be very complex, because programs are typically funded as
stand- alone programs, not as comprehensive packages. As distinct programs, they are ad-ministered
by different areas within an agency and within different agensies and have differ-ent
billing and reporting requirements. In addition, although goals, services, and target popu-lations
overlap to some degree, programs can be implemented in sufficiently different ways
to make coordination difficult. The OWCH should conduct a formal study to assess the feasi-bility
of comprehensively coordinating prenatal outreach and early childhood health ser-vices.
The OWCH has already studied coordination of home visitation programs as part of
the Arizona Family Preservation/ Family Support State and Local Plans.
RECOMMENDATION
1. The ADHS should conduct a formal study to assess the feasibility of comprehensive
program coordination that includes the following:
a. A comprehensive listing of all prenatal/ early childhood health outreach and pre-vention
programs, including state, local, federal, and county programs.
b. An assessment of related programs' goals, type of intervention, availability through-out
the State, and costs.
c. An assessment of the needs of target populations, target communities, and current
community resources meeting these populations' needs.
d. An analysis of various consolidation strategies, to determine how program con-solidation,
consolidation of different programs' administration, or block granting
could help the OWCH to improve comprehensive service delivery.
FINDING Ill
HEALTH START REVERTS A SIGNIFICANT
AMOUNT OF ITS APPROPRIATION
An estimated $ 500,000,' about 50 percent, of the 1994- 95 appropriation will revert to the
State General Fund, with much of the reversion attributed to funds budgeted for direct
client services during the fiscal year. The portion of expenditures spent on services was
low in 1994- 95 because the contract period covered only four months and because the
OWCH established reimbursement requirements that precluded paying contractors for
services until the participant's baby is born. To avoid reversion in the future, the Legisla-ture
may want to consider alternative methods for funding new pilot programs.
Short Contract Period
Due to delays in the contracting process, Health Start was able to serve participants only
for the last four months of 1994- 95. Legislation authorizing the Health Start pilot was
passed in June 1994, and funds were available to the OWCH by September 17,1994, but
actual service did not begin until March 1995. Although the time from RFP distribution to
contractor selection took only two months, negotiating rates for each contractor and au-thorizing
contracts took up to six months to complete.
Some procurement delays can be expected with the start of a pilot program; however,
many can be avoided by negotiating rates expeditiously. Although many of the pilot-funded
Health Start contractors are not new, the Program did increase the number of
participants served and expanded the services offered. Similarly, because contractors were
required to follow new policies and procedures and negotiated multiple rates for a vari-ety
of services offered, contract negotiation was time- consuming. In comparison, the
Healthy Families Pilot Program was also constrained by many of these same pilot- year
issues but was still able to provide additional months of service in its first year. In Cochise
County, where both Health Start and Healthy Families programs are run by Cochise County
Health and Social Services, the Healthy Families contract was effective on November 7,
1994, and the program started screening participants on February 1,1995. In contrast, the
Health Start contract was not effective until March 1,1995, and the program did not start
serving participants until June 14,1995. In the future, Health Start should take steps to
provide service in a more timely manner.
The estimate of funds that will revert is based on the budget balance for all revenue sources after
payment of allbills that the OWCH showed as " approved for payment" as of November 21,1995, and
auditor estimates on the amounts of late administrative adjustments made against fiscal year 1994- 95
funds.
Reimbursement Policy
Health Start instituted a policy that delays payments to contractors until after the baby is
born. For example, if a woman's baby is due in February 1996, a contractor who provides
prenatal services in June 1995 would not be reimbursed until after the baby is born. Con-sequently,
expenses that are incurred in one fiscal year might not be paid until the next
fiscal year. This practice will result in some service funds reverting to the State General
Fund.
Alternatives to Reversion
Requiring a program to revert remaining funds to the General Fund after the pilot
program's first year may be counterproductive. Program administrators may not be able
to estimate accurately how much service can actually be provided in the first year. Fur-ther,
it may take longer than a year to develop programs in needy areas.
Two alternatives could be considered when funding first- year programs. One alternative
is to provide only program planning and development funds in the first year. This would
reduce reversions and allow time to develop programs in areas where services do not
already exist.
A second alternative is to make first- year funding non- revertible. This allows programs
to use funding for direct services in the second year, if there are delays in implementing
the program. The reversion clause for the Family Literacy Pilot Program, created by the
same legislation as Health Start, was eventually removed by the Legislature. The At- Risk
Preschool Expansion, also created during the 1994 9th Special Session, is non- revertible
and did not spend any of its fiscal ? ear 1994- 95 appropriation until September of 1995.
RECOMMENDATION
For future pilot programs, the Legislature should consider using one of two alterna-tive
funding methods:
provide only program planning and development funds in the first year, or
W make first year funding non- revertible to allow funds to be used for direct services in
the succeeding year.
FINDING IV
ARIZONA FAMILY RESOURCE GUIDE
MEETS GOALS
The Arizona Family Resource Guide was developed to aid parents in accessing needed ser-vices
without relying on public programs for referrals.' The Guide has been well devel-oped
and fulfills its goal of increasing parents' access to information.
Background
The purpose of the Arizona Family Resource Guide is to enable parents to access informa-tion
concerning the needs of their young children without relying on public programs,
such as Health Start and Healthy Families. The Guide was mandated to list private and
public providers of services relating to early childhood development and family support.
The Arizona Fanzily Resource Guide is a " directory of directories." It lists organizations that
can either provide the information or services necessary, or can refer the consumer to the
appropriate providers in that geographic area. It is a wallet- sized folding card that lists 51
resource and referral organizations and their telephone numbers ( mostly toll- free num-bers).
The Guide is available in English and Spanish, and both versions indicate whether
or not the organizations have Spanish- speaking staff available. The organizations are
grouped by 5 topics - emergency'numbers, family services ( child care; child and water
safety; counseling and emotional support; family support and financial help; and infor-mation
and referral), education, health, and special needs. ( See Appendix C for an illus-tration
of the Guide.)
As of July 25,1995, the OWCH had distributed almost 320,000 copies of the Guide. About
176,000 copies were distributed, in accordance with legislative mandate, to hospitals; the
Healthy Families Pilot Program; and the Health Start contractors. The remaining copies
were distributed to agencies listed on the Guide, county health departments, other public
and private programs and service providers, various advisory committees, ADHS ad-ministrators,
and the Legislature. All copies have been distributed free of charge.
The Arizona Children and Families Stability Act, which created the Health Start Pilot Program, also
directed the ADHS to develop and distribute a new directory called the Arizona Children and Families
Resource Directo y. The Arizona Family Resource Guide is the name of the guide developed by the OWCH
in response to this mandate.
Well- Developed Guide
Meets Its Goals
The Guide has been well developed and should fulfill its goal of increasing parents' ac-cess
to information and referrals. The Guide is based on a thorough review of program
needs, and is both usable and informative. It has only a few minor problems that should
be remedied before it is reprinted.
Guide preparation based on tlzorough review of needs - In preparing the Guide, the
OWCH conducted a thorough review of provider and consumer needs. An external con-tractor
made recommendations about the Guide's format and content, based on a review
of existing resource directories in Arizona and interviews with staff from state- funded
hospitals, ADHS staff, and staff from other programs. The contractor also obtained input
from OWCH staff in identifying topics to cover and the specific organizations that needed
to be listed.
The review showed that family needs would best be met by a small directory of organiza-tions
that provide information and referrals over the telephone. Referral information from
these providers is more likely to be accurate and complete than it would be in a printed
directory, and is also more accessible to parents who have limited reading skills, do not
speak English, or have special needs. Several sources have noted that it is very expensive
to maintain current and complete directory information. Since a number of organizations
already maintain this information, it is much cheaper to refer individuals to these organi-zations
than it would be to duplicate their efforts.
The resource guide's size gives it several practical advantages over a large printed direc-tory
of all resources. Since it is portable, it is more likely that it will be carried and used. It
saves storage space, a major concern of hospitals. Finally, it saves the State money. The
OWCH initially budgeted $ 80,000 for the first printing of a directory ( which they had
initially planned as a printed booklet); yet its actual printing cost for the wallet cards was
only $ 12,026, at 3 cents per copy.
TIre yrintedgziide is usable and i~ zfonizative- The printed card is easy to use and refers
consumers to organizations that can provide needed information or referrals. To assess
the Guide's usability, Auditor General staff called resource and referral services listed to
ask for referral information and to determine hours the organizations were available.
Through these telephone calls, the Office of the Auditor General staff attempted to iden-tify
( a) any inaccuracies and ( b) any barriers ( such as difficulty reaching a human voice,
or long periods of time spent on hold) that would potentially discourage parents from
using the guide. When we called, we found the staff to be helpful and able to provide the
information we requested. In addition, we found that assistance was easily accessible
over the telephone: busy signals were rare and most calls were answered within 5 rings.
The first contact with the organization was generally with a person, and time spent on
hold was limited.
Several minor problems wed to be corrected - Although the Guide is accurate and useful
overall, several minor problems may hinder parents and other consumers wishing to use
it. The main barrier encountered was that some organizations did not have Spanish- speak-ing
staff available to take the call, even though they were listed as having Spanish speak-ers
available. Other problems were limited, but should be addressed. Two telephone num-bers
were not current and need to be updated. The Spanish version contained several
typesetting problems including inconsistencies with the English version and the omis-sion
of one resource and referral service. Finally, the Guide does not indicate all the orga-nizations
with 24- hour telephone accessibility, making it difficult for consumers who work
diirjng tiie day to & jlize r; es" iirces- w- iiha fkr- hours avaiiabiiij ,
Before the Guide is reprinted, the OWCH should contact each listed organization to verify
and update the following information: ( a) that the telephone number is correct, ( b) that
they regularly have Spanish speakers available on staff, ( c) that their staff are prepared to
provide referrals in their topic area, and ( d) the hours that telephones are staffed. The
OWCH should also add a symbol identifying numbers with 24- hour accessibility, and
add cross- references.
RECOMMENDATION
1. Before the next printing, the OWCH should verify and update information listed on
the Guide. The OWCH should also add a symbol identifying organizations that are
available 24 hours.
STATUTORY
ANNUAL EVALUATION COMPONENTS
Session Laws 1994, 9th Special Session, Chapter 1, Section 9 instructs the Office of the
Auditor General to include ten factors in annual program evaluations of the Health Start
Pilot Program. Responses to these factors are listed below.
1. Information on the number and characteristics of the program participants.
A total of 2,843 women were considered " active" in the Health Start database dur-ing
the four- month service period of March 1,1995, through June 30,1995. Of these
potential participants, 2,7401 were enrolled in Health Start and 103 received out-reach
but did not enroll in the Program.
More than two- thirds of the women active in the Program are of Hispanic back-ground.
The race/ ethnicity distribution of these women is:
H Asian/ Pacific Islander - .2%
African- American/ Non- Hispanic - 7.5%
H Native American - 13.1%
White/ Non- Hispanic - 9.5%
H Hispanic - 69.8%
More than half of the women who enrolled in the program already had health care
through the Arizona Health Care Cost Containment System ( AHCCCS); however,
a substantial percentage have no health insurance, as shown below:
H Enrolled in AHCCCS - 54%
H Applied for AHCCCS - 5.9%
H Private Insurance/ Care - 3.1%
H No Insurance - 37.1%
These participants include new prenatal participants and participants from the previous program
who continued family follow- up under the new program. Some of the family follow- up participants
did not receive visits that were billed during the four- month period, but were still considered to be
active in the Program.
Fifty- two percent of the women are in their twenties and another 18 percent are 30
and older. Almost half of the women are married.
2. Information on contractors and program service providers.
The BWCH contracted with 13 providers for Health Start services in over 50 urban
and rural sites across Arizona.' These contractors include six county health depart-ments
( one of which is providing services through a subcontract to a private pro-vider)
and seven private, not- for- profit providers ( four health centers, a behav-ioral
health center, and two area health education centers). Five providers have
served metropolitan areas in Phoenix, Tucson, and Yuma, and 10 contractors have
served rural areas in 11 of Arizona's 15 counties. Two of the contractors ( one ur-ban,
one rural) serve primarily Native American participants.
Table 1 ( see Introduction and Background, page 6), shows the contractors, their
service areas, service expenditures in 1994- 95, and their pilot program contract
award for 1995- 96. Most contracts were funded for a 16- month period, including
the last 4 months of fiscal year 1994- 95 and fiscal year 1995- 96.
3. Information on program revenues and expenditures.
State, federal, and private funds have been combined to develop and administer a
single Health Start Pilot Program. State funds come from the pilot program fund-ing
( Arizona Children and Families Stability Act) and an existing state appropria-tion
for Health Start/ Prenatal Outreach. Other sources include an allocation from
a federal block grant for Maternal and Child Health ( MCH) and private grants
from the National Association for the Education of Young Children ( NAEYC). The
1994- 95 revenues totaled $ 1,365,584. Revenues for 1995- 96, totaling $ 1,636,695, are
shown in Figure 1 ( see Introduction and Background, page 5). This figure includes
$ 1.6 million in state dollars plus a $ 36,695 allocation from the 1995- 96 MCH block
grant; however, the final allocation from the MCH block grant may change over
the course of the year.
One contractor ( a private ~ rovider) c anceled their contract duing 1994- 95, leaving 12 current con-tractors.
Health Start expenditures from all revenue sources totaled $ 819,762' in 1994- 95.
Sixty- six percent of expenditures were spent on services to clients, 6 percent was
spend on the Arizona Family Resource Guide and training, and 28 percent went to
administrative costs. An estimated 50 percent of the 1994- 95 Arizona Children and
Families Stability Act Health Start appropriation will revert to the State General
Fund. A more detailed discussion of the reasons for reversion of funds is included
in Finding 111 ( see pages 18 through 19).
4. lnformation on the number and characteristics of enrollment and disenrollment.
For the four- month Health Start service period of March 1,1995, through June 30,
1995, 2,740 clients were considered to be active participants in the Program. An
additional 103 women were contacted by Health Start outreach, but did not enroll
in the Program. Of these nonparticipants, 79 were not pregnant and 24 declined
participation.
Of the 2,740 clients enrolled during the four- month service pedod, 156 were inac-tive
as of June 30,1995. Of these 156,99 were inactive because they moved and 47
withdrew from the program. Eight women miscarried and two refused further
services.
The marital status/ living situation, age distribution, and race/ ethnicity of women
who are no longer active in the Program do not appear to differ significantly from
women who are still active in the Program.
5. lnformation on the average cost for each participant in the program.
The estimated average cost per participant for the first four months of service to
participants was $ 728. This estimate includes four- month service costs for these
participants and all first- year administrative costs except costs for developing and
printing the Arizona Family Resource Guide. ( See Appendix D, page a- xvii, for the
methods used in estimating the cost per participant.) Participants received an av-erage
of 2.77 visits, yielding a rate of $ 263 per visit. If administrative costs ( salaries
and benefits, travel, professional and outside services, CATS, equipment, and other
operating expenses) are excluded, the average service cost perparticipant is $ 401,
yielding a rate of $ 145 per visit.
We estimate that the final 1994- 95 expenditure total will increase slightly once pending invoices and
administrative adjustments to 1994- 95 funds clear the system.
The true cost to serve a participant over the Program's full length ( prenatal plus
four years of family follow- up) cannot be estimated at this time. Although the
extended service period will add to the cost per participant, first- year start- up and
administration costs will amortize over participants enrolled in future years, tak-ing
away from the cost per participant. Appendix D ( see page a- wii) describes the
procedures used in estimating the cost per participant.
6. Information concerning progress of program participants in achieving goals
and objectives.
This report does not address the progress of participants in achieving program
goals and objectives. Information demonstrating progress toward program goals
should be available in the 1996 and 1997 evaluation reports, after new participants
have been enrolled in the Program long enough to expect to see progress in achiev-ing
goals and objectives. However, the Office of the Auditor General has some
concerns about evaluating the outcomes of the Health Start Pilot Program.
First, although the Program's stated goals conform to the law, the individuals imple-menting
it ( both OWCH staff and the providers) do not agree on the measurable
outcomes one would expect to see as a result of those goals. For example, even
though the Program provides training on health behaviors, some program staff do
not believe the Program's effectiveness should be evaluated by measurable changes
in these behaviors.
Second, the Program may be unable to show impact on at least two of eight statu-tory
goals: reducing the incidence of low birth weight babies and increasing pre-natal
care services to pregnant women. Due to site and participant selection, many
of the communities selected to receive Health Start do not have rates of low birth
weight that exceed state or county averages, and a few do not have rates of inad-equate
prenatal care that exceed state or county averages. It will be very difficult to
show that the Program has improved these rates if it targets communities having
better than statewide rates. Also, the Program does not have individual eligibility
criteria other than being pregnant. Although this may fit the program model, it
allows participants with less need to enter the program if they wish. For example,
at the time they entered Health Start, 54 percent of the participants active in the
pilot program during March though June 1995 were already enrolled on AHC-CCS
and an additional 3 percent had private insurance for prenatal health care.
Another 6 percent had already applied for AHCCCS at the time they enrolled in
Health Start. While enrollment in AHCCCS does not guarantee use of prenatal
care, at least some of these women were probably in prenatal care before they
enrolled in Health Start. If the program enrolls many participants like this over the
thee years, it will be very difficult to show its impact on access to prenatal care.
Third, the program collects limited evaluative information on participants. The
current forms collect information primarily about the services that were delivered
to participants. For example, they do not collect participant knowledge of health
care or use of the health care system. They also collect limited information about
participant demographics that would show that the Program is reaching the ap-propriate
target population. We acknowledge that lay health workers are not trained
to do formal assessments and that participants may find the forms to be intrusive.
However, the scarcity of information on program participants limits the scope and
comprehensiveness of the outcome evaluation.
7. Recommendations regarding program administration.
a. The OWCH should revise their evaluation criteria and methods in the follow-ing
ways ( see Finding I, pages 10 through 13):
W Establish several variables under which a community can qualify as having
high need.
Allocate more points for need in a proposal's overall score.
W Apply criteria more consistently in evaluation proposals.
W Consider identifying specific sites with the highest need and requesting pro-posals
specifically to serve those communities.
b. The ADHS should conduct a formal study to assess the feasibility of compre-hensive
program coordination that includes the following four elements ( see
Finding 11, pages 14 through 16):
A comprehensive listing of all prenatal/ early childhood health outreach
and prevention programs, including state, local, federal, and county pro-grams
W An assessment of related programs' goals, type of intervention, availability
throughout the State, and costs
W An assessment of the needs of target populations, target communities, and
current community resources meeting these populations' needs
An analysis of various consolidation strategies, to determine how program
consolidation, consolidation of different programs' administration, or block
granting could help the OWCH to improve comprehensive service delivery.
8. Recommendations regarding informational materials distributed through the
programs.
Recommendations regarding the Arizona Children and Families Resource Directory
are covered under item # 10 below. The Auditor General's Office has no other rec-ommendations
regarding materials distributed through the Program at this time.
9. Recommendations pertaining to program expansion.
The Office of the Auditor General has no recommendations pertaining to program
expansion at this time.
10. Recommendations regarding the method used in preparing the Arizona Chil-dren
and Families Resource Directory.
Before the next printing, the OWCH should verlfy and update information listed
in the Guide ( see Finding IV, see pages 20 through 22). The OWCH should add
a symbol identifying organizations that are available 24 hours and add cross-references.
R Agency Response I
I Office of the Director
m 1740 W. Adams Street FIFE SYMINGTON, GOVERNOR
Phoenix, Arizona 85007- 2670 JACK DILLENBERG, D. D.. S, M . P. H., D IRWJTOR
( 602) 542- 1025
( 602) 542- 1062 FAX
January 24, 1996
I Mr. Douglas R. Norton, CPA
Auditor General
Office of the Auditor General
2910 North 44th Street, Suite 410
Phoenix. Arizona 85004
Dear Mr. Norton:
Thank you for the opportunity to review the revised preliminary report of the first Annual
Evaluation of the Health Start Pilot Program and the Arizona Children and Families Resource
Directory. I am proud of the efforts of the Health Start Team and welcome your
acknowledgment that we served a total of 2,843 women during the first four service months.
u While I appreciate the amount of time given to this effort, I continue to have some concerns
about the evaluation team's understanding of public health and community based programs.
Evaluating community based programs is a very challenging responsibility. The Department is
I committed to working with your staff in identifying the best evaluation methodologies.
Please know that the department will continue to analyze thoroughly your comments and
- Leadership for a Healthy Arizona -
AUDITOR GENERAL'S ANNUAL EVALUATION OF THE HEALTH START PROGRAM:
Arizona Department of Health Services Response
Thank you for the opportunity to comment on your review of the Health Start
Program. The Arizona Department of Health Services is proud of the Health Start
Team and welcomes your acknowledgment that we served 2,843 women during the
first four months of service and that the Arizona Children and Families Resource
Directory was well designed and implemented.
G HEALTH START TARGETS HIGH RISK COMMUNITIES
ADHS would like to emphasize that in the selection of pilot sites, evaluation criteria
were followed by the nine individuals who reviewed and scored the proposed sites.
We are confident that all of the more than 60 communities that are being served
through 13 Health Start contractors are appropriate matches for Health Start. These
include economically disadvantaged urban neighborhoods in Phoenix and Tucson,
migrant farm worker communities, Native American reservation areas and isolated
rural communities throughout Arizona.
Evaluation Criteria Included Many Factors
The selection of all Health Start sites was based on a determination of need that was
not limited to birth weight and prenatal care statistics. As a basis for determining
need, ADHS used all four of the criteria specified in the Arizona Children and Families
Stability Act, plus additional criteria. That act ( Chap. 1 of the 9th Special Session of
the 1993- 94 Legislature) said the evaluation criteria " shall include at a minimum a
high incidence of inadequate prenatal care, infant health care, low birth weight babies
or inadequate early childhood immunizations." All of the proposals accepted for
funding demonstrated need based on at least one of the four indicators.
ADHS also looked at a community's teen pregnancy and poverty rates, as well as
availability of medical services to further demonstrate need. The Request for
Proposals specifically required prospective contractors to submit a statement
documenting the unmet health needs of pregnant women, infants and children in the
communities they were proposing to serve. Appropriate evaluation criteria were used
and those criteria were followed. We believe that a review of the communities being
served speaks for itself ( see page 4 of this response).
Community Capacity Critical to Success
We agree with the recommendation that more weight could be given to community
ADHS Response to the AG1s Health Start Evaluation January 1996 Page 2
need in future reviews of proposals. It is important to note that ADHS sought to
implement the legislative expansion of Health Start as expeditiously as possible.
Therefore the capacity of a community was an important concern. To ensure timely
program start- up, communities were awarded points in scoring for displaying capacity
by: 1) submitting a plan for performing required services; 2) demonstrating
experience; 3) highlighting qualifications, training and education of personnel; and 4)
demonstrating knowledge of their community and support from their community.
Regarding the recommendation to identify sites with the highest need first and then
request proposals to serve them, this would not only have delayed implementation,
it would not be consistent with local initiative and decision making.
At the outset, the Department could not put a Health Start Program in every
potentially qualifying community. ADHS will continue to work with those
communities through other methods in order to improve their capacity to respond to
future program opportunities.
0 HEAL TH START ENHANCES PROGRAM COORDINA TION
Since its inception, program coordination has been accepted as an essential
component of Health Start and is addressed at both the policy and operational levels.
From an operational standpoint, the lay health worker is the key to program
coordination for the client. This local health professional is knowledgeable of all the
services available in the community and is therefore best able to assist the client to
select and access those programs which best meet the family's needs.
The lay health worker, in turn, is trained by the local Health Start agency whose
responsibility it is to coordinate with other programs in the area. ADHS is strongly
committed to the development of local infrastructures which are capable of tailoring
broadly designed state and federal health and social programs to meet local needs.
ADHS also is working diligently at the policy level to enhance coordination of
programs which provide similar services andlor target similar populations. These
efforts are taking place not only internally but in conjunction with other state
agencies, community based organizations, and local parent groups. Issues being
addressed include: variations in eligibility requirements, eligibility procedures, program
goals, types of interventions, and distribution of service providers.
Since so much is under way to enhance program coordination, ADHS believes a
formal study would not be an effective use of resources and in actuality could serve
ADHS Response to the AG's Health Start Evaluation January 1996 Page 3
to delay change. The points suggested by the Auditor General for consideration in
a feasibility study already are being addressed at the program, agency, and legislative
levels and will continue to be the focus of efforts to make programs more efficient.
Q HEAL TH STAR T FUNDS SPEN P EFFECT1 VEL Y
The Health Start Program was fully operational for only four months of FY 1995.
Consequently, ADHS reverted a substantial amount of money because of
reimbursement policies and the short service delivery period. This will be the only
year that this level of reversion will occur. It should be noted that despite the short
time frame, the program served more than 2,800 people.
The delay in program start- up was the result of system modifications. With the
enactment of the Family and Children Stability Act, ADHS had to make significant
changes to the scope of its existing Health Start Program. This necessitated the
creation of new policies and procedures and the solicitation of bids for the new
services as existing contracts could not be adequately modified.
ADHS designed contractor reimbursement policies to encourage quality services.
Local Health Start agencies are paid for services after a client's baby is born. This
reimbursement policy is based on the concept that prenatal care and good maternal
health practices relate to healthy newborns.
We concur with the Auditor General's recommendation that the Legislature consider
alternative means for funding new pilot programs.
4 HEALTH START RESOURCE GUIDE MEETS GOALS
The Health Start Family Resource Guide is an excellent tool for anyone trying to
locate education, health, and/ or social service programs for families. Although small
enough to fit in a wallet, the Guide is packed with information. This directory,
available in both English and Spanish, lists organizations that either provide direct or
referral services. Further, the information is grouped by topic to make it easier to
figure out where to go for assistance.
ADHS appreciates the Auditor General's excellent evaluation of the Family Resource
Guide. We concur with the recommendation to verify and update the information,
and to indicate which agencies provide 24 hour service.
ADHS Response to the AG's Health Start Evaluation January 1996 Page 4
Q HEALTHSTARTMEETS LOCAL NEEDS
Health Start has been implemented as a community based program. ADHS is
confident that this model was the legislative intent. This means that locally the
program reflects the unique characteristics of the community as a whole and the
individuals who live there. This presents very real challenges in evaluating the
program. Program staff will continue to work with the evaluators in designing and
implementing an evaluation appropriate for this model. We are confident that valid
measures can be found to demonstrate efficacy.
For example, indicators have already been used to show community progress. The
number of prenatal visits, low birth weight rates and immunization rates are outcome
measures that the program is collecting. The following is an overview of information
available regarding women in the Health Start Program who gave birth between
March I, 1995 and November 30, 1995, their infants, and other children ( up to age
2) in the family during this period.
* Prenatal Care Gains Reported
Of the 668 women delivering during this time period, 6.3% had fewer than five
prenatal visits; 25.2% had 5- 8 visits; and, 63.6% had 9 or more visits. Therefore,
the inadequate prenatal care rate of 6.3% for women in the program is lower than the
state rate of 7.4% ( when including no care). This is a very important finding because
one of the criteria used for selecting these communities and neighborhoods was their
high rates of inadequate prenatal care. The program will continue to collect and
analyze these data to monitor improvement over time.
* Low Birth Weight Rate Drops
There were 30 babies born weighing less than 5 pounds, 8 ounces. This results in
a low birth weight rate of 45 per 1,000 live births for this high risk population which
is well below the state rate of 67. Since the numbers are small, the significance of
the differences will be more meaningful as the program progresses over time.
* immunization Rates Improve
1,182 children received services during this time period. 52.7% had received
immunizations appropriate for their age. This is a significant increase from the
baseline penetration rate; as of February 1995 only 11% were appropriately
immunized. It is expected that the immunization rates will continue to improve as the
program continues. Because of the need to space the administration of certain
ADHS Response to the AG's Health Start Evaluation January 1996 Page 5
vaccines it takes several months to complete the series required for adequate
immunization levels.
In summary, it appears that the program is meeting established goals. The targeted
numbers are being reached and the data is demonstrating program efficacy and cost
effectiveness.
Appendix A
Health Start
Proqram Activities and Anticipated Outcomes for Service Periods
Note on Table: Bold items in the right column indicate anticipated outcomes that are primary outcome goals of the program.
Outreach: Activities
Identify pregnant woman needing services residing in service area . Assist woman in obtaining a pregnancy test
W Describe program
Explain rights and responsibilities of client and lay health worker
Explain informed consent
Enroll woman if she is pregnant and requests enrollment
Prenatal Period: Activities
W Assist client in accessing prenatal care from a medical provider, preferably
in the first trimester
W Help woman enroll on AHCCCS or find other way of paying for prenatal
care and delivery
W Provide basic prenatal and perinatal education
W Assist client in overcoming barriers to care
Assist client in accessing financial assistance if appropriate
W Provide referrals for client or other family members to other community
resources, as appropriate
Family Follow- up Period: Activities
W Encourage mother to have all her children fully immunized at the
appropriate ages
W Educate families about good nutritional habits
Educate client about preventive health care and child wellness
W Assist with finding a primary source for receiving routine medical care
(" medical home") for each family member . Review prenatal and perinatal topics as indicated
H Educate mother about the importance of having children screened for
hearing and vision, assessed for developmental disabilities, etc.
W Assist client in applying for private and public financial assistance
H Distribute Arizona Family Resource Guide
W Assist client in accessing adult services, including education, employment
and other community involvement, etc.
. H A ct as role model for client, as a client advocate . P romote positive parenting skills Encourage mother to enroll her children in preschool programs
Source: Auditor General staff analysis of Health Start enabling legislation, interviews with OWCH staff and its program coordinators,
review of program materials, and review of literature.
Anticipated Outcomes
N/ A
. Anticipated Outcomes lmprove neonatal health
outcomes, including birth
weight . Increase in families receiving
assistance, if eligible.
. Anticipated Outcomes Decrease rates of childhood
disease
W Improve overall health of
children
ldentify health problems
( vision, hearing, etc.) or
childhood disabilities early
Increase in families receiving
assistance, if eligible.
8 Improve client's ability to
access any needed services
independently
lnciease client independence
and decrease reliance on
. p ublic assistance Decrease child abuse
W Improve child's chances of
good academic performance
a- iii
Appendix B
Programs with Goals or Services
that Parallel Health Start
Programs Sharing Some of Health Start's
Prenatal Goals or Services
Baby Arizona
Community Health Advisor Training Project
Coordinated Care ( under AHCCCS Plan)
Healthy Mothers, Healthy Babies
Indian Health Services Public Health Nursing
Opening Doors ( Havasupai, Hualapai reservations)
Pregnancy and Breastfeeding Hotline
Prenatal Care Initiative ( Tucson)
Project Cumadre ( areas in Pinal County)
Support for Obstetrical Services
Teen Prenatal Express
Wellness on Wheels/ Rural Health Outreach ( areas in Yavapai County)
Woman to Woman ( Pima County)
Children's Information Center
Community Health Advisor Training Project
Community Nutrition Education Services
Coordinated Care ( under AHCCCS Plan)
Community Health Nursing
First Steps
Healthy Families
Healthy Mothers, Healthy Babies
Indian Health Services Public Health Nursing
Newborn Intensive Care Program
Opening Doors ( Havasupai, Hualapai reservations)
Pregnancy and Breastfeeding Hotline
Preventive Nutrition Services
Prenatal Care Initiative ( Tucson)
Project Chance
Project Cumadre ( areas in Pinal County)
Project Thrive
Woman to Woman ( Pima County)
Wellness on WheelslRural Health Outreach ( areas in Yavapai County)
Primary Funding
Source
State
Private
State
Federal
Federal
Private
Federal
County
Federal
State
State
Federal
Private
Program Sharing Some of Health Start's Primary Funding
Post- NatallFamilv Follow- Up Goals or Services Source
State
Private
State
State
State
Private
State
Federal
Federal
State
Private
Federal
State
County
Federal
Federal
State
Private
Federal
Source: Auditor General staff analysis of interviews with program coordinators from Health Start and
other programs; Health Start pilot program proposals, ADHS Community and Family Health
Services Annual Report, 1994.
a- vii
Appendix C
Copy of Arizona Family Resource Guide
( English Version)
Diroicr W: Smu Cn4 Cochiu, 03um &
Orenkc Cnmda + 1- 440- 216- 719
\ EMERGENCY \
IEALm drtcs doclor & nunc & Ira, child
hedry Q v b i i I.& luby # boo b. o h
M& rcr** a.
L EDUCATION h FAMILY SERVICES
CHILD CARE
M G xRa urce & kfcml+ 1600I. C8- W
CAMlLY SWPORT L FINANCIAL W
Behaviont Health Au&
Behaviwrl Hcdth SLnicadYwu ( BHS):
L I h & Y U I N W + 1- 1
Community Pvmcnho 6r bhniml
Hc. lrhCn ( carc. rr) + 1600d) l. I314
rillmdfyRmrnhiplaSoudum~
( CPSA):~, GnkOmnk, Rau
QSMCMCardn + 1440.281- 9189
Nadvm Ackau k w bhniml
Hedth Audvriq ( NARBHA): Apck,
Cocaaim,. . h.. id.. w.. e.,. . N...+ . io & Y 4 Counda 1 ~ Z I U
R N m 8 BclUvbnl Hc& Avoddan
( FG. B. H.. A.).: . G. l. 8. .&.. R. r. u. l + C 1a4u4\ 0d. aW 2.1317
Arizona
Family Resource Guidc
cknmlmiry rn& lnudon rd k& d
NonhernbrCenarlMm~ ................. + 14.4.0... 3.3.2.-. 3.7.9.2
ldanucion.. d.... &... l. e. d.. S+ cr r1i4c0e& r SWou- Mthe7r4n
a- xi
Appendix C
Copy of Arizona Family Resource Guide
( Spanish Version)
............ W E U k * IQDIOIJAl
.-.. a.. C.. r.. L.. b.. m.. c..-..+.. n.....+.. w..... I... s i tI4m4s4lU
WIY. CIYrlrh% ba" a
A h . . . . . ..................... * I Q c y c l I 3
w n............~.... *~ aoa rr~ u
. W... Y..... U.. 6... M...&.. a.. L................ ... t 1- 1OY
muv t -.& h..-.. Q... l.. r., A... ... ...... 1- tYnn
Cula de Recursos para
la Familia en Arizona
para Familias con Nilla
Pequellos y Hiia eon
Necesidades F. spccii1~ 1
de Cuidado de la'Wud
EN EMERCENCIAS L EDUCAC~~ N SERMCIOS A FAMILUS SERVlClOS A FAMW h i -
1 SAUID
a- xiii
~ R q k u k ~ U h d ~
~ ~ s . r r * r . l Y m ( U 0 1 :
M . . a b h r L Y n u . . . . , t l ~
trmu* lh.. M...). l. r. b.. r. W... r. W...- cn 1- l- 1314
Comuyp-- wbr- k( l. rr
m-- Gnhm-
Ck s - CM t.. ...... 1awat4im
mFAMIMESP ARA
Appendix D
Procedures Used to Estimate Cost per Participant
Two estimates of Health Start cost per participant have been computed. The first
estimate includes only direct service delivery; the second estimate includes direct service
delivery and administrative expenditures.
Table 3
Health Start Cost Per Participant Estimates
for Fiscal Year 1994- 95
A. Direct Service Delivery Costs
( March 1, 1995, through June 30, 1995)
B. Administrative Costs for 1994- 95
( July 1,1994, through June 30,1995),
excluding the Arizona Family Resource Guide
C. Total Cost
D. Total Number of Participants
E. Direct Service Cost Per Participant - ( A/ D)
F. Total Cost Per Participant - ( C/ D)
" Cost per participant estimates were based on a four month service delivery period from March
through June 30, 1995. The number of participants used for this estimate ( 770) is smaller than the
number reported in item 1 of the Statutory Annual Evaluation Components section of this report,
becaue it is based on the number of participants billed, not the number of participants active in the
Program during the time period. An additional 1,123 prenatal participants received services that were
not billable in 1994- 95 because their babies had not been born by the end of the fiscal year. Another
950 participants were considered to be active in the family follow- up portion of the Program, but did
not receive home visits that were billable during 1994- 95.
Source: Direct service delivery costs and total number of participants were calculated using OWCH data
for services billed March 1, 1995, through June 30, 1995. Administrative costs were calculated
based on expenditure data provided by ADHS Business and Financial Services.
a- xvii