Evaluation, Perinatal Substance Abuse Pilot Program report to the Arizona Legislature

              State of Arizona
Office
of the
Auditor General
PERFORMANCE AUDIT
Report to the Arizona Legislature
By Debra K. Davenport
Auditor General
PERINATAL
SUBSTANCE ABUSE
PILOT PROGRAM
November 2001
Report No. 01-31
The Auditor General is appointed by the Joint Legislative Audit Committee, a bipartisan committee
composed of five senators and five representatives. Her mission is to provide independent and impar-tial
information and specific recommendations to improve the operations of state and local government
entities. To this end, she provides financial audits and accounting services to the state and political
subdivisions and performance audits of state agencies and the programs they administer.
The Joint Legislative Audit Committee
Senator Ken Bennett, Chairman
Representative Roberta L. Voss, Vice-Chairman
Senator Herb Guenther Representative Robert Blendu
Senator Dean Martin Representative Gabrielle Giffords
Senator Peter Rios Representative Barbara Leff
Senator Tom Smith Representative James Sedillo
Senator Randall Gnant (ex-officio) Representative James Weiers (ex-officio)
Audit Staff
Carol Cullen—Manager
and Contact Person (602) 553-0333
Beth Vogl—Team Member
Copies of the Auditor General’s reports are free.
You may request them by contacting us at:
Office of the Auditor General
2910 N. 44th Street, Suite 410
Phoenix, AZ 85018
(602) 553-0333
Additionally, many of our reports can be found in electronic format at:
www.auditorgen.state.az.us
2910 NORTH 44th STREET • SUITE 410 • PHOENIX, ARIZONA 85018 • (602) 553-0333 • FAX (602) 553-0051
DEBRA K. DAVENPORT, CPA
AUDITOR GENERAL
STATE OF ARIZONA
OFFICE OF THE
AUDITOR GENERAL
WILLIAM THOMSON
DEPUTY AUDITOR GENERAL
November 1, 2001
Members of the Arizona Legislature
The Honorable Jane Dee Hull, Governor
Ms. Catherine R. Eden, Director
Department of Health Services
Transmitted herewith is a report of the Auditor General, An Evaluation of the Perinatal
Substance Abuse Pilot Program administered by the Arizona Department of Health
Services. This evaluation was conducted pursuant to Laws 1998, Ch. 176, §3. I am also
transmitting with this report a copy of the Report Highlights for this evaluation to
provide a quick summary for your convenience.
As outlined in its response, the Department of Health Services plans to implement all of
the recommendations.
My staff and I will be pleased to discuss or clarify items in the report.
This report will be released to the public on November 2, 2001.
Sincerely,
Debra K. Davenport
Auditor General
Enclosure
i
OFFICE OF THE AUDITOR GENERAL
SUMMARY
The Office of the Auditor General has completed an evaluation
of the Perinatal Substance Abuse Pilot Program, pursuant to
Laws 1998, Ch. 176, §3. This pilot program uses an integrated
model for administering services to women who use or are at
risk of using substances, and who are either pregnant or have
recently given birth. Under this approach, steps are taken to in-volve
providers of medical, behavioral, and social services in a
collaborative network. The program’s purpose is to improve ser-vice
providers’, or collaborators’, ability to address the health
and well-being of mothers and their children by sharing infor-mation
and developing a more coordinated set of services. The
Legislature has appropriated $83,000 annually to fund the inte-grated
program for fiscal years 2000 through 2002. An additional
$200,000 was appropriated to the program in 2000 from Tempo-rary
Assistance for Needy Families (TANF) funds. The pilot pro-gram,
called EMSA (Expectant Mothers with Substance Abuse)
Esperanza, has been operating since July 1999 and is located in
Tucson, Arizona. It enrolled 67 women between November 1999
and May 2001. The program terminates in June 2002.
Program Needs to
Improve Integration of
Services
(See pages 13 through 19)
While the pilot program has implemented some elements of an
integrated services program, it needs to make additional im-provements.
In an integrated program, clients should have ac-cess
to comprehensive medical, behavioral health, and social ser-vices.
These services should be coordinated among the service
providers, or collaborators, through sharing client information in
a central location. To help it achieve integration, the pilot pro-gram
has taken some steps, such as conducting regular meetings
with collaborators. As a result of these meetings, collaborators
indicated that they have increased their knowledge of available
community resources.
Summary
ii
OFFICE OF THE AUDITOR GENERAL
However, the pilot program has not sufficiently implemented
other important elements of integration. Specifically, collabora-tors
were expected to use a shared information system to track
client needs, services received, and demographic information.
Sharing such information would allow the collaborators to
jointly manage and coordinate services for clients, thereby en-hancing
the possibility of a positive outcome for substance-abusing
mothers and their children. However, collaborators did
not effectively share information through this system. Further,
the information sharing about clients that did occur at monthly
collaborator meetings could also be improved. Discussions about
clients generally did not lead to the co-management or develop-ment
of plans for the clients’ care. Finally, the pilot program lacks
a formal leadership mechanism to develop program direction
and address barriers to service integration, such as concerns with
sharing confidential client information. The pilot program in-tends
for its Advisory Board to make recommendations for pro-gram
improvement, but has proposed minimal ways to gather
information so that the Board can develop appropriate policies
and strategies.
If the Legislature decides to fund the pilot program beyond June
2002, or if it continues without state funding, program staff need
to make improvements. Specifically, the staff need to develop
methods to obtain client information and effectively share it with
all collaborators. If the pilot program continues to use monthly
meetings to discuss client care, the meetings should be used as a
forum for co-managing client care and integrating services. Fi-nally,
the pilot program should continue with its plan to expand
its Advisory Board and charge the Board with the responsibility
for helping to develop program direction and address barriers to
service integration.
Program’s Impact
Cannot Be Assessed
(See pages 21 through 27)
Although the law creating the program requires a report on the
outcomes of the program, including whether clients achieved a
Summary
iii
OFFICE OF THE AUDITOR GENERAL
drug-free status, evaluators were not able to accomplish this be-cause
they lacked data and had other limitations. Evaluating a
program’s impact requires knowing which services clients re-ceived
and what happened as a result. In this pilot program,
both types of information are insufficient. For example, while the
program has information about the number of times that clients
were referred for services, it does not have adequate information
about whether these services were actually provided. Informa-tion
on outcomes is also missing. The program did not properly
monitor client drug use and relapse rates. It also did not consis-tently
monitor the women’s general health.
Information is available on the health status of 23 babies whose
mothers gave birth to them while participating in the program,
but the health status cannot be attributed to the mother’s partici-pation
in the program. This is because too few women entered
the program at each stage of pregnancy and birth information
was not available for all women who gave birth while in the
program. Although research suggests that a baby benefits no
matter when the mother stops using drugs, positive birth out-comes
are more likely to occur if a woman enters care in her first
trimester.1 However, only 9 of the 67 women entered the pro-gram
during their first trimester.2 Further, birth information is
available for only 23 women. To draw reliable conclusions about
the program’s impact, sufficient birth information must be avail-able
for a sufficient number of mothers entering the program at
each stage of pregnancy.
If the program continues, action is needed to improve record-keeping,
develop ways to ensure that the outcomes identified in
the program’s enabling law are measured, and bring women
into the program earlier in their pregnancies.
1 Monjaraz, Connie. A Study of the Relationship of Early Prenatal Care to Birth
Weight. Does First Trimester Care Make a Difference? University of Ne-braska—
Omaha, 2001.
2 Information on date of enrollment into the program and estimated
date of delivery is available for 61 clients. Information is not avail-able
for the remaining 6 clients.
iv
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
v
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS
Page
Introduction and Background.......................... 1
Finding I: Program Needs to
Improve Integration of Services.................. 13
Participation in an Integrated
Services Program............................................................. 13
Program Has Taken Some
Steps To Achieve Integration.......................................... 14
Increased Information Sharing
Needed for Further
Integration....................................................................... 15
Actions Can Be Taken
To Increase Information Sharing.................................... 18
Recommendations .......................................................... 19
Finding II: Program’s Impact
Cannot Be Assessed.................................... 21
Extent to Which Clients
Receive Services Is Unknown......................................... 21
Program Did Not Collect Sufficient Data
Needed To Assess Outcomes......................................... 23
Too Few Clients in
Program at Each
Stage of Pregnancy.......................................................... 25
If the Program Continues, Improvements
Are Needed To Establish Outcomes.............................. 26
Recommendations .......................................................... 27
Table of Contents
vi
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS (Concl’d)
Page
Statutory Evaluation Components ................... 29
Agency Response
Figures
Figure 1 Perinatal Substance Abuse Pilot Program
Pilot Program Oversight and Administration
July 1999 through May 2001......................... 3
Figure 2 Perinatal Substance Abuse Pilot Program
Stage of Pregnancy Women Entered
the Program
November 1999 to May 2001........................ 25
Tables
Table 1 Perinatal Substance Abuse Pilot Program
Schedule of Revenues and Expenditures
Years Ended or Ending June 30, 2000,
2001, and 2002 (Unaudited).......................... 6
Table 2 Perinatal Substance Abuse Pilot Program
Type, Number, and Percentage of
Service Referrals
November 1999 to March 2001..................... 22
1
OFFICE OF THE AUDITOR GENERAL
INTRODUCTION AND BACKGROUND
The Office of the Auditor General has completed an evaluation
of the Perinatal Substance Abuse Pilot Program. The evaluation
was conducted pursuant to Laws 1998, Ch. 176, §3. This report
provides information on program administration, implementa-tion,
and outcomes, and offers recommendations for improve-ment.
Pilot Program Is Designed
To Improve Health and
Welfare Through an
Integrated Set of Services
Through Laws 1998, Ch. 176, §5, the Legislature created the Peri-natal
Substance Abuse Pilot Program (pilot program), which is
based on an integrated model for delivering services to pregnant
or postpartum women who are substance abusers or are at risk
for substance abuse. The program received an original appro-priation
of $83,000 per year for fiscal years 2000 through 2002.
The integrated model was developed by a legislatively author-ized
committee charged with developing a statewide strategy for
addressing substance abuse by women during or after preg-nancy.
1 This committee’s recommendation for an integrated
model was made in a September 1997 report, Community-Based
Integrated Model for Pregnant and Parenting Substance Abusing
Women.
The primary goal of an integrated approach is to improve the
health and welfare of pregnant or parenting substance-abusing
women and their children through better access to a combination
of medical, behavioral health, and social services. In a noninte-grated
approach, service providers operate independently of
1 Laws 1995, Ch. 215, §1 created the Advisory Council on Perinatal Sub-stance
Abuse, known more recently as the Implementation Oversight
Committee on Perinatal Substance Abuse.
The program serves preg-nant
or postpartum
women with substance
abuse problems.
Introduction and Background
2
OFFICE OF THE AUDITOR GENERAL
each other. For example, if a pregnant woman seeks obstetrical
care, a medical provider typically focuses on the mother’s medi-cal
needs. In an integrated service model, providers from each
system coordinate and share information about medical and be-havioral
health care and social services for the woman and her
family.
The integrated program’s specific goals are to:
¾ Expand the target population’s knowledge of and access to
services;
¾ Improve birth outcomes and client and family well-being;
¾ Ensure that substance-abusing women have access to com-prehensive
health care;
¾ Reduce unnecessary duplication of case management and
improve efficiency in the service delivery system;
¾ Maximize existing resources for clients through collaborative
partnerships; and
¾ Increase providers’ service awareness.
Pilot Program Organization,
Staffing, and Funding Sources
The Department of Health Services administers the program—
The Department of Health Services (Department) developed the
request for proposals and administers the pilot program through
a contract with Community Partnership of Southern Arizona, the
Department’s Regional Behavioral Health Authority (RBHA) in
Pima County.1 Community Partnership of Southern Arizona
1 A Regional Behavioral Health Authority (RBHA) is a community organi-zation
that administers behavioral health services in the State of Arizona.
They contract with providers to deliver a full range of services, including
prevention, substance abuse, and general mental health programs.
Introduction and Background
3
OFFICE OF THE AUDITOR GENERAL
Figure 1
Perinatal Substance Abuse Pilot Program
Pilot Program Oversight and Administration
July 1999 through May 2001
(Contractor)
Community Partnership
of Southern Arizona
(Subcontractor)
El Rio Health Center
(Pilot Program)
Expectant Mothers
with
Substance Abuse
Oversight
Committee
Interagency
Service
Agreement
Workgroup
Advisory
Board
Collaborators
(Administrative Agency)
Department of Health Services
(DHS)
Source: Auditor General staff summary of program oversight and administration contained in the program
contract and legislation.
Introduction and Background
4
OFFICE OF THE AUDITOR GENERAL
subcontracted this program to El Rio Health Center in Tucson,
Arizona. The pilot program, which is called EMSA (Expectant
Mothers with Substance Abuse) Esperanza, was started in July
1999 and enrolled its first client in November 1999.
To put this program into effect, Laws 1998, Ch. 176, §2 estab-lished
the Interagency Service Agreement Workgroup (work-group).
The workgroup is composed of a representative from the
Department of Health Services, the Department of Economic Se-curity,
the Arizona Health Care Cost Containment System, and
the Governor’s Community Policy Office. The workgroup meets,
generally on a monthly basis, to address issues such as identify-ing
services that can assist pregnant substance-abusing women
and encouraging collaboration among service providers.
Pilot program staffing—
When the program began, it had 1.75 positions as follows:
¾ Director (.5 FTE)—Provides program oversight.
¾ Coordinator (.5 FTE)—Maintains relationships with com-munity
agencies and helps decrease barriers to a woman’s
access to services. El Rio Health Center funds began provid-ing
salaries for an additional .5 coordinator FTE as of May
2000.
¾ Computer Specialist (.5 FTE)—Maintains the Health Pro
database, connects collaborators to the database, and pro-vides
training on the system.
¾ Clerk (.25 FTE)—Assists with customer service and keeps
records.
In fiscal year 2001, the program added an additional position:
¾ Assistant (1.0 FTE)—Conducts initial assessments, makes
referrals, ensures a woman receives services, and maintains
data collection.
Introduction and Background
5
OFFICE OF THE AUDITOR GENERAL
The pilot program also has an Advisory Board, which is typically
composed of representatives from up to three agencies. The
Board’s purposes include defining program principles, identify-ing
available community services, and addressing service barri-ers.
The Board has been meeting quarterly since April 2000.
Pilot program has been appropriated about $450,000 over 3 fis-cal
years—Program appropriations have come from two sepa-rate
sources:
¾ Tobacco Tax funds ($249,000)—In 1998, the Legislature
(Laws 1998, Ch. 176, §5) appropriated to the Department of
Health Services $83,000 per year for 3 years (2000-2002) for
this program. The money was allocated from Tobacco Tax
funds for program implementation, including staff salary,
computer equipment, and other office supplies and expenses.
These funds could be used for integration activities, but not
for new or expanded direct services or case management.
¾ Temporary Assistance for Needy Families funds (TANF)
($200,000)—In 2000, the Legislature (Laws 2000, Ch. 393,
§17) made an additional one-time appropriation of $200,000
to this program from the TANF program. In deciding how to
apply this additional money, the workgroup approved the
pilot program’s plans to spend it on salaries for additional
staff, transportation, and various services, including legal as-sistance,
home-based support, and day care vouchers. The
additional monies cannot be used to fund medical treatment
(A.R.S. §46-300.04) (see Table 1, page 6) but can be used for
case management. See Statutory Evaluation Components,
pages 29 through 34, for further information about this ap-propriation.
The program’s original
appropriation was
$83,000 per year for 3
years.
Introduction and Background
6
OFFICE OF THE AUDITOR GENERAL
Pilot Program Has
Several Key Components
Three important components of the program, given its collabora-tive
model, are interagency participation, an assessment for ser-
Table 1
Perinatal Substance Abuse Pilot Program
Schedule of Revenues and Expenditures
Years Ended or Ending June 30, 2000, 2001, and 2002
(Unaudited)
2000 2001 20024
(Actual) (Actual) (Estimated)
Revenues:
Tobacco Tax and Health Care Fund
appropriation 1 $83,000 $ 83,000 $ 83,000
Temporary Assistance for Needy Families
(TANF) block grant 28,571 2 171,429
Total revenues $83,000 $111,571 $254,429
Expenditures 3 $83,000 $111,571 $254,429
1 Consists of monies transferred from the Fund’s Medically Needy Account administered by the Arizona
Health Care Cost Containment System in accordance with Laws 1998, Chapter 176, §5.
2 Consists of monies reimbursed from the Department of Economic Security in accordance with A.R.S. §46-
300.04.
3 Amounts shown are payments made or estimated to be paid to Community Partnership of Southern
Arizona. The 2001 amount includes $28,571 owed to the Partnership, but not paid at June 30, 2001.
4 The Department of Health Services believes the Partnership will spend at least $83,000. However, TANF
monies are paid retroactively based on subcontractor performance. Consequently, the Department is
unable to estimate 2002 TANF revenues and expenditures. Amounts shown are equal to remaining
authorized TANF monies and appropriated Tobacco Tax and Health Care Fund monies.
Source: Auditor General analysis of the Arizona Financial Information System’s Revenues and Expenditures
by Fund, Program, Organization, and Object report for the years ended June 30, 2000 and 2001; and
the State of Arizona Companion Transaction Entry/Transfer form dated July 30, 2001.
Introduction and Background
7
OFFICE OF THE AUDITOR GENERAL
vices, and an online information-sharing system that can provide
information to all participating agencies.
¾ The program operates through agency collaboration—
Eight organizations have signed memos of agreement with El
Rio Health Center to participate in the pilot program. As
stipulated in the agreement, these formal collaborators agree
to refer appropriate women to the pilot program, provide
services for clients, and participate in networking meetings
and outreach efforts. The eight formal collaborators are as fol-lows:
  Medical Facilities (3)—Two facilities, El Rio Obstetrics
and Gynecology and El Rio Midwifery, offer medical ser-vices
for women throughout their participation in the pi-lot
program. A third, the Rural Health Office’s mobile
health clinic, provides medical care for individuals who
may not otherwise have access to health care. Commu-nity
health advisors from this agency contact pregnant
women in their homes or on the street to inform them
about the clinic’s services and to provide prenatal educa-tion.
  Substance Abuse Treatment Facilities (3)—Two col-laborators,
CODAC Las Amigas and The Haven, are
residential substance abuse treatment centers. Women
typically live at the centers for 6 to 12 months.1 They re-ceive
substance abuse counseling, life-skills training, and
case management, and in some instances, have their chil-dren
living with them on-site. A third facility, La
Frontera, operates a methadone maintenance clinic,
where clients receive daily doses of methadone. The
length of treatment depends on the individual client, with
an average length of 3 years.
  Domestic Violence Agency (1)—Brewster Center is a
domestic violence facility that offers shelter, outreach, le-gal
assistance, and food boxes.
1 Women can stay at The Haven for up to 6 months and at CODAC Las
Amigas for up to 12 months.
There are eight formal
collaborating agencies
from the medical, behav-ioral
health, and social
services fields.
Introduction and Background
8
OFFICE OF THE AUDITOR GENERAL
  Homeless Drop-in Shelter (1)—At Casa Paloma, a
homeless drop-in shelter, women receive basic necessities
such as food, showers, and laundry facilities. The shelter
also maintains some bed space for women who are not
currently using substances. These women can stay in the
residence for up to 2 years.
In addition to the 8 formal collaborators, 24 agencies participate
in the program as informal collaborators. They, too, offer services
to pilot program clients, but have not signed memos of agree-ment
to share information about a client’s care. These agencies
include representatives from the medical, behavioral health, and
social service fields, as well as Native American tribes, a school
district, and legal counsel.
¾ Eligible women complete an intake assessment—To be-come
enrolled, a woman must meet the following criteria:
  Pregnant or up to 1 year postpartum,
  Enrolled in AHCCCS 1or eligible to enroll, and
  Using drugs or at risk of using drugs, or
  At risk of losing child custody due to drug use.
In addition to determining if a woman meets these eligibility cri-teria,
she receives an intake assessment, conducted by the pro-gram
coordinator or assistant, to determine the services she
needs. This assessment addresses client and family demograph-ics,
drug use history, and the woman’s need for different types of
services; for example, medical, drug treatment, transportation,
legal, or housing. The woman is then referred to various provid-ers
offering services. Pilot program staff or providers may make
additional referrals. The program has a policy that requires the
assessment to be updated every 3 months to determine if addi-tional
services are needed.
1 The Arizona Health Care Cost Containment System (AHCCCS) is the
State's Medicaid program.
Introduction and Background
9
OFFICE OF THE AUDITOR GENERAL
n Program has an information system for facilitating
communication among collaborating agencies—Formal
collaborators are to use an online information-sharing sys-tem,
Health Pro, to share information and to facilitate com-munication
about the woman’s care within the integrated
services system. Only formal collaborators who have signed
memos of agreement are eligible to use the system. This da-tabase
allows collaborators to track client referrals and ser-vices
and follow various aspects of a woman’s care.
Characteristics of Women
in the Pilot Program
Between November 1999 and May 2001, 120 women were re-ferred
to the program by 25 different medical, behavioral health,
or social service agencies. Forty-five percent of the women were
referred by medical providers, 27 percent by behavioral health
providers, and 26 percent by social service agencies. The remain-ing
clients (2 percent) were referred to the program from other
sources. Many of the 120 women did not meet eligibility re-quirements
or complete the intake assessment, or were not en-rolled
in the program for other reasons. Thus, a total of 67
women were enrolled in the pilot program from November 1999
through May 2001.
The typical client has given birth to two children, is Hispanic, 29
years of age, not married, unemployed and currently not seeking
employment, and has less than a high school education. Further,
upon entry into the pilot program, the typical client reported
having used multiple types of drugs and was in her third trimes-ter
of pregnancy. See Statutory Evaluation Components, pages
29 through 34, for further details about the program and its cli-ents.
Evaluation Scope and
Limitations
Laws 1998, Ch. 176, §3 calls for this evaluation to assess specific
outcomes. However, due to a lack of data and other limitations,
evaluators were unable to assess the pilot program in accordance
The program enrolled 67
women by May 1, 2001.
Introduction and Background
10
OFFICE OF THE AUDITOR GENERAL
with the criteria set forth in statute. Evaluators were to assess the
following specific outcomes:
¾ Drug Usage and Well-Being—
a. Successful strategies for reducing or eliminating sub-stance-
abusing behaviors,
b. Number of months a woman is drug-free,
c. Relapse rates, and
d. Status of woman’s and family’s well-being.
Evaluators did not have the necessary information to assess
these outcomes. Evaluators worked with the program staff to
establish appropriate methods of collecting such information.
These methods include establishing a baseline of conditions
when a woman enters the program and measuring a
woman’s drug use history and general well-being at 3-month
intervals, and at program completion. However, as discussed
in Finding II (see pages 21 through 27), program staff and col-laborators
did not adequately collect this information.
¾ Infant Drug Status at Birth—Information is available on the
health status of 23 babies whose mothers gave birth to them
while participating in the program, but the health status can-not
be attributed to the mother’s participation in the pro-gram.
This is because too few women entered the program at
each stage of pregnancy and birth information was not avail-able
for all women who gave birth while in the program. Al-though
research suggests that a baby benefits no matter
when the mother stops using drugs, positive birth outcomes
are more likely to occur if a woman enters care in her first
trimester.1 However, only 9 women entered the program
during their first trimester, 10 entered during their second
trimester, and 19 women entered during their third trimester.
Further, birth information is available for only 3 women who
entered during in their first trimester, 4 who entered in their
second trimester, and 16 who entered in their third trimester.
To draw reliable conclusions about the program’s impact,
sufficient birth information must be available for a sufficient
1 Monjaraz, Connie. A Study of the Relationship of Early Prenatal Care to Birth
Weight. Does First Trimester Care Make a Difference? University of Ne-braska—
Omaha, 2001.
Due to insufficient pro-gram
data, outcomes
could not be assessed as
required by the Legisla-ture.
Introduction and Background
11
OFFICE OF THE AUDITOR GENERAL
number of mothers entering the program at each stage of
pregnancy.
Methods
This report presents findings and recommendations in two areas:
¾ The program needs to improve its integration of services.
¾ Although many service referrals have been made, the pilot
program did not collect sufficient data needed to assess out-comes.
A variety of methods was used to assess the pilot program’s abil-ity
to integrate services. The evaluators made 21 site visits to the
pilot program from November 2000 through April 2001. During
these site visits, evaluators attended collaborator and Advisory
Board meetings, observed client meetings, and conducted inter-views
with program administrators, formal collaborators, and
informal collaborators. Further, evaluators completed a file re-view
of 16 of the 67 client files to verify the accuracy and com-pleteness
of the Health Pro database. Intake assessment informa-tion
collected in the Health Pro database from November 1999 to
May 2001 was also analyzed to determine client and child traits,
drug use and treatment history, and referrals made for the 67 en-rolled
clients.
Additionally, evaluators observed 12 Oversight Committee
meetings and 12 workgroup meetings from July 1999 to May
2001. Finally, evaluators documented the process by which the
contractor and subcontractor were selected.
Acknowledgements
The Auditor General and staff express appreciation to the De-partment
of Health Services, the Community Partnership of
Southern Arizona, the El Rio Health Center, and the pilot pro-gram
agencies and clients for their cooperation and assistance
throughout the evaluation.
12
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
13
OFFICE OF THE AUDITOR GENERAL
FINDING I PROGRAM NEEDS TO
IMPROVE INTEGRATION
OF SERVICES
While the pilot program has implemented some elements of an
integrated services program, it needs to make additional im-provements.
Components of integration include a client receiv-ing
comprehensive services and service providers, or collabora-tors,
sharing client information in a central location. To help it
achieve integration, the pilot program has taken some steps, such
as conducting regular meetings with the collaborators. However,
the pilot program has not sufficiently implemented other ele-ments
of integration that it proposed, such as effective methods
of sharing client information. The pilot program should make
additional improvements to further integrate services and in-crease
information sharing.
Participation in an Integrated
Services Program
In an integrated services program, a pregnant or parenting sub-stance-
abusing woman should be able to simultaneously access a
variety of medical, behavioral health, or social services that she
needs. The pilot program, in its proposal, stated that it would
establish links among collaborators by providing them with a
common database information system. This system would allow
collaborators to easily co-manage a woman’s various needs. The
resulting communication regarding client care among collabora-tors
would enable them to track client services, and enhance the
possibility of a positive outcome for substance-abusing mothers
and their children.
The pilot program proposed the following process for a client’s
participation:
¾ A woman appears for services at the pilot program;
Finding I
14
OFFICE OF THE AUDITOR GENERAL
¾ The program coordinator or assistant determines if the
woman is eligible for the program;
¾ If eligible, the woman enrolls in the pilot program and re-ceives
an assessment to identify her service needs;
¾ The program staff refer the woman to collaborators for
needed medical, behavioral health, or social services;
¾ Collaborators use the pilot program’s shared information
system to obtain client demographic, assessment, and referral
data; and
¾ As the woman receives care, the collaborator provides infor-mation
within the database system about the referrals and
the outcome of the services and makes additional referrals as
necessary.
The pilot program also intended that the collaborators and pro-gram
staff would meet regularly to discuss the services available
to clients. Further, program staff would ensure that collaborators
make referrals and provide services, and that the collaborators
use the shared information system to effectively coordinate the
clients’ care.
The pilot program also established an Advisory Board whose
members were to meet regularly to resolve program implemen-tation
barriers. Potential barriers could include different ap-proaches
on how to work with women; organizational problems,
such as a lack of support from top management; conflicting data
requirements; or legal issues regarding confidentiality of shared
information.
Program Has Taken Some
Steps To Achieve Integration
After the pilot program’s contract was approved in July 1999, the
pilot program took some steps to help it achieve integration.
Specifically, it:
¾ Enrolled 67 women from the 120 referrals received.
Finding I
15
OFFICE OF THE AUDITOR GENERAL
¾ Enlisted the participation of 8 formal and 24 informal col-laborators.
Formal collaborators have signed memorandums
of agreement to enter information in the shared information
system, provide services for pilot program clients, and par-ticipate
in monthly meetings with program staff. Informal
collaborators have not signed such agreements and therefore
do not have access to the shared information system. How-ever,
they do provide services to pilot program clients and
can attend monthly meetings with program staff.
¾ Conducted 18 monthly meetings from September 1999 to
April 2001. An average of 4 formal collaborators and 3 infor-mal
collaborators attended the meetings, ranging from 3 to 11
participants at any meeting.
¾ Provided community service information, including presen-tations
from five local service providers, during the monthly
meetings.
¾ Prepared a “gaps analysis” of needed community resources.
According to interviews with formal collaborators, their partici-pation
in the pilot program has helped increase their knowledge
of available community resources.
Increased Information Sharing
Needed for Further
Integration
While the pilot program has taken several steps to achieve inte-gration,
it has yet to sufficiently implement some important ele-ments
of an integrated program. First, although the pilot pro-gram
proposal and contract call for sharing client information
through the Health Pro information system, providers are not
sharing information and the system is not being used as in-tended.
Second, the client information sharing that does occur at
monthly meetings could be more effective. Finally, the pilot pro-gram
lacks a formal leadership mechanism for developing pro-gram
direction.
Finding I
16
OFFICE OF THE AUDITOR GENERAL
Information-sharing system is not being used—Although the
pilot program’s proposal stated that collaborators were to estab-lish
“integrated care through communication and clinical path-ways
among each other using a comprehensive management
information system called Health Pro,” the system is not being
used as intended. Health Pro enables collaborators to obtain cli-ent
information including basic demographics, intake and as-sessment
information, and service referrals, and to manage client
care. However, client information has not been consistently
shared through Health Pro. First, the majority of the service pro-viders
did not become formal collaborators and therefore did not
have access to Health Pro. Twenty-four of the 32 collaborators
are informal participants and they account for 84 percent of all of
the pilot program’s referrals for service. Because informal col-laborators
cannot use the data system, information on these re-ferrals
must be obtained by program staff who then enter it into
the database. This was not done consistently.
Second, even the formal collaborators did not fully use Health
Pro. They did not consistently provide information to the data-base,
nor did they find the information that was available in
Health Pro to be useful. Collaborators gave various reasons for
not providing or using information in Health Pro, including the
burden of additional work, not having an accessible way to pro-vide
information, and concerns about confidentiality of informa-tion
contained within the system. However, by not consistently
using the information system, collaborators could not determine
a woman’s service needs or whether a woman received services
from other providers.
The pilot program, realizing that there were problems with
Health Pro, proposed that its use be made optional, and identi-fied
alternative ways (fax, e-mail, or courier) in which informa-tion
would be shared. However, the pilot program has not speci-fied
how client information will be shared among collaborators
using those means. Unless the pilot program develops specific
procedures for sharing information about a woman’s care, it
cannot improve service integration. Further, the pilot program
should clarify how these alternative methods will reduce dupli-cative
service efforts, which is one goal of an integrated ap-proach.
Finding I
17
OFFICE OF THE AUDITOR GENERAL
Ineffective client information sharing occurred at collaborator
meetings—Although pilot program staff and collaborators dis-cussed
specific client cases during monthly meetings, these dis-cussions
were not effective in coordinating a woman’s care. Dur-ing
these meetings, information about a client, such as her preg-nancy,
living situation, or progress in treatment, was shared
among the group. However, evaluators attended meetings be-tween
September 1999 and March 2001, and did not identify any
instances in which the discussion led to the group co-managing
the client’s care, developing an action plan, establishing timelines
to accomplish specific tasks, or following up on clients at subse-quent
meetings. When the collaborators do not perform these
functions, the responsibility falls upon the program’s limited
staff resources. For example, a formal collaborator shared that:
“…one of the clients is homeless, lives in the de-sert,
and is going to deliver her baby soon. The
community health worker has taken the woman
to her pre-natal appointments, but is concerned
that the baby will be removed from the woman’s
care because of her living situation. [The collabora-tor]
stated that she is not sure where the woman
should go, what type of services she can receive,
or has received through [the pilot program].”
Participants in the meeting suggested places where the woman
could receive services, but they did not offer to assist with the
woman’s care or establish a plan for coordinating her services.
Instead, at the end of the discussion, the program coordinator
offered to work directly with the collaborator to assist this client.
Pilot program lacks a formal leadership mechanism for develop-ing
its direction— Even though written agreements are used to
formalize the collaboration, the pilot program lacks a mechanism
to further develop implementation plans. Collaborating agencies
have agreed to provide services, but the pilot program does not
have a formal means for addressing the barriers that may occur
in integrating services, such as concerns about confidentiality of
client information. Although the pilot program calls for expand-ing
its Advisory Board and charging it with providing recom-mendations
for program improvement, it has proposed minimal
Finding I
18
OFFICE OF THE AUDITOR GENERAL
ways to gather information necessary to develop appropriate
policies and strategies. Additionally, the Board met only four
times in almost 2 years and only had one to three individuals at-tend
each meeting. Thus, it was unable to guide program im-plementation
or identify available community resources.
Actions Can Be Taken
To Increase Information Sharing
In the absence of a database information system that can be used
to effectively share information, steps can be taken to ensure that
client information is still communicated among collaborators. If
the Legislature decides to fund the pilot program beyond June
2002, or if the pilot program continues without state funding,
program staff need to make specific improvements in the follow-ing
areas:
¾ Develop an approach for sharing client information—
Program staff need to develop methods to obtain client in-formation
and make it available to all collaborators as
needed. The information-sharing methods should ensure cli-ents
are getting needed services and help reduce duplicative
service provision. Once the communication methods are es-tablished,
the pilot program should revise its memorandums
of agreement to help ensure that formal collaborators use the
methods to consistently share client information.
¾ Use monthly meetings more effectively—Collaborators
can make more effective use of the monthly meetings to inte-grate
services and manage client care. Specifically, the meet-ings
could be used as a forum for co-managing client cases
and sharing community resource information. If the pilot
program continues to use meetings to discuss client cases,
program staff should ensure that collaborators develop ac-tion
plans for client care with timelines and follow up with
client case presentations.
¾ Develop a formal leadership mechanism—The pilot pro-gram
should continue with its plan to expand the Board’s
membership and charge the Board with responsibility for
helping to develop program direction and address barriers to
Finding I
19
OFFICE OF THE AUDITOR GENERAL
service integration. The pilot program has already contacted
representatives from a variety of medical, behavioral health,
and social service agencies and requested their participation
on the Board. The Board should meet quarterly and obtain
information from both collaborators and pilot program cli-ents
to help develop program direction and address barriers.
For example, the Board should address concerns with shar-ing
confidential client information in monthly meetings if the
pilot program continues to use those meetings to discuss cli-ent
care.
Recommendations
1. The pilot program should develop methods to share client
information with collaborators and revise the memorandums
of agreement accordingly.
2. If the pilot program continues to discuss client cases during
the monthly meetings, program staff should ensure that col-laborators
develop action plans for client care with timelines
and provide follow-up presentations at future meetings.
3. The pilot program should continue to recruit Advisory Board
members. The Board should meet on a regular basis to ad-dress
barriers to service integration and should use client and
provider information to make recommendations for program
improvement.
20
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
21
OFFICE OF THE AUDITOR GENERAL
FINDING II PROGRAM’S IMPACT
CANNOT BE ASSESSED
The law creating the program requires a report on the outcomes
of the program, including whether clients have achieved a drug-free
status and whether there has been an improvement in the
health and well-being of the clients and their infants. However,
the data needed to make such a report is not available. The pro-gram
currently does not have sufficient data on the services that
clients actually receive, or on changes in the clients’ drug use and
health status. Although data is available on birth outcomes for
some clients’ infants, there are too few women who entered the
pilot program at each stage of pregnancy for evaluators to assess
the program’s impact on these outcomes. While the program has
proposed corrective actions to address these problems, its actions
so far are incomplete.
Extent to Which Clients
Receive Services Is Unknown
One important aspect of assessing the program’s outcomes is de-termining
which services clients received; however, the pilot
program does not have sufficient information. Although the
program has information showing the extent to which clients
were referred for services, it is not clear how many of these refer-rals
resulted in services. A total of 364 referrals for services had
been made for women in the program between November 1999
and March 2001—an average of 6 referrals for each client.1 See
Table 2 (page 22) for the types of referrals and number made.
When a woman is referred to a formal collaborator, the collabo-rating
agency is responsible for indicating in the Health Pro da-tabase
that the woman did or did not receive a service. If the
woman is referred to an informal collaborator, then the program
1 Referrals are those made between November 1999 and March 2001. Re-cords
showed that 57 of the 67 clients had been referred for services; 10
had no referrals as of March 2001.
Referrals are made but
extent to which services
are received is unknown.
Finding II
22
OFFICE OF THE AUDITOR GENERAL
coordinator is responsible for making these notations. According
to a pilot program administrator, these updates should be done
within 1 month of referral.
However, a key problem is that some of the Health Pro database
entries do not contain enough information to determine if the
service was received. When a referral is updated in the Health
Pro database, the database calls only for making a notation that
the referral is “closed.” However, a notation that the referral is
service was received. When a referral is updated in the Health
“closed” does not necessarily mean that a service was received.
Rather, a sufficient entry requires entering additional informa-tion
in the case notes.
Table 2
Perinatal Substance Abuse Pilot Program
Type, Number, and Percentage of Service Referrals
November 1999 to March 2001
Type of Referral Number
Percentage of
Total Referrals
Substance abuse treatment 52 14.3%
Parenting 50 13.7
Housing 35 9.6
Government assistance 32 8.8
Medical 28 7.7
Mental health 22 6.0
Employment 22 6.0
Education 21 5.8
Legal aid 19 5.2
Clothing/baby items 19 5.2
Transportation 17 4.7
Nutrition 16 4.4
Child care 15 4.1
Counseling/advocacy 9 2.5
Case management 5 1.4
Domestic violence 2 .6
Total referrals 364 100.0%
Source: Auditor General staff analysis of service referrals made for 57 pilot
program clients from November 1999 to March 2001.
Finding II
23
OFFICE OF THE AUDITOR GENERAL
Program Did Not
Collect Sufficient Data
Needed To Assess Outcomes
Although the law creating the program requires a report on sev-eral
different client-related outcomes, the pilot program did not
collect sufficient data required to determine these outcomes.
Outcomes specified in the law include whether the woman
achieved a drug- and alcohol-free status and, if so, whether the
woman subsequently returned to substance-abusing behaviors.
Other outcomes include the status of the woman’s and the fam-ily’s
general health. However, the program did not collect
needed data regarding these outcomes. For example, the pro-gram
did not monitor clients’ drug use and did not consistently
follow-up on the clients’ general health.
The pilot program did not monitor client drug use or relapse—
Drug use and relapse rates cannot be reported because the pro-gram
did not systematically monitor a client’s drug use. To de-termine
substance use and relapse rates for women in the pro-gram,
drug use should be monitored in an objective manner and
on a regular basis. Child Protective Services, the courts, and col-laborating
substance abuse treatment centers do so through
urine analysis testing. Urine analysis testing can determine
changes in the pattern, frequency, and amount of an individual’s
drug use.
Because the original program funding could not be used for
urine analysis testing, the pilot program agreed to collect self-reported
drug use information at 3-month intervals, although
this is less reliable method for monitoring drug use.1 However,
the pilot program did not conduct the proper number of client
follow-up assessments. Evaluators’ review of client cases as of
March 2001 revealed that none of the clients received the proper
1 Preston et al. Comparison of Self-Reported Drug Use with Quantitative
and Qualitative Urinalysis for Assessment of Drug Use in Treatment: The
Validity of Self-Reported Drug Use: Improving the Accuracy of Survey Estimates.
NIDA Research Monograph, 167, 1997, pages 130-144.
Finding II
24
OFFICE OF THE AUDITOR GENERAL
number of follow-up reports. Because the pilot program did not
monitor client drug use, evaluators cannot determine if the
women achieved a drug-free status. Further, pilot program ad-ministrators
are not able to adequately monitor the program’s
effectiveness relative to this outcome.
The program did not consistently monitor a client’s health or
well-being—Although the pilot program collected general health
and well-being information when a client entered the program,
this information was not regularly updated, meaning that any
change in the woman’s health and well-being could not be read-ily
determined. A woman’s need or receipt of services is deter-mined
through the use of the Integrated Services Tool (IST),
which is administered by program staff at intake and should be
done again every 3 months. Services addressed on the IST are:
Medical Case Management Child Care
Perinatal Transportation Education Training
Mental Health Financial Counseling Employment
Substance Abuse Income Eligibility Parenting Skills
Domestic Violence Housing CPS
Nutrition Dependent Care Other Child Welfare
However, the program did not consistently follow up with each
active client to determine the current status of her general health
and well-being. Only 16 percent of the required follow-up inter-views
were conducted; thus, consistent updates of client cases in
the Health Pro system were not done. Since the program did not
adequately monitor the woman’s general health or well-being,
formal collaborators cannot view updates of what each woman
needs; pilot program administrators do not have necessary in-formation
about the clients; and evaluators are unable to deter-mine
if any improvement has been achieved.
Finding II
25
OFFICE OF THE AUDITOR GENERAL
Too Few Clients in
Program at Each
Stage of Pregnancy
Information is available on the health status of 23 babies whose
mothers gave birth to them while participating in the program,
but the health status cannot be attributed to the mother’s partici-pation
in the program. This is because too few women entered
the program at each stage of pregnancy and birth information
was not available for all women who gave birth while in the
program. Although research suggests that there may be benefits
for the baby no matter when the mother stops using drugs, posi-tive
birth outcomes are more likely to occur if a woman enters
care in her first trimester.1 However, only 9 women entered the
program during their first trimester, 10 entered during their sec-ond
trimester, and 19 women entered during their third trimes-ter.
Further, birth information is available for only 3 women who
1 Monjaraz, Connie. A Study of the Relationship of Early Prenatal Care to Birth
Weight. Does First Trimester Care Make a Difference? University of Ne-braska—
Omaha, 2001.
Figure 2
Perinantal Substance Abuse Pilot Program
Stage of Pregnancy Women Entered the Program
November 1999 to May 2001
After the
baby is
born (23)
28 to 40
weeks (19)
14 to 28
weeks (10)
Up to 14
weeks (9)
Source: Auditor General staff analysis of 61 clients’ dates of enrollment into the pilot
program and their estimated dates of delivery. Information was not available
for six clients.
Finding II
26
OFFICE OF THE AUDITOR GENERAL
entered during their first trimester, 4 who entered in their second
trimester, and 16 who entered in their third trimester. To draw
reliable conclusions about the program’s impact, sufficient birth
information must be available for a sufficient number of mothers
entering the program at each stage of pregnancy. Finally, 23
women entered the program after giving birth, so the program
could not have had any impact on their birth outcomes. See
Statutory Components, pages 29 through 34, for birth informa-tion.
If The Program Continues,
Improvements Are Needed To
Establish Outcomes
After evaluators called these problems to the attention of the re-sponsible
parties, the program developed proposals for address-ing
the problems in a corrective action plan. If the program is
continued, further actions are needed to ensure that outcomes
can be measured and achieved.
Corrective action plan acknowledges need to make improve-ments—
The corrective action plan includes proposals for im-plementing
urine analysis testing for the clients, capturing other
outcome data on a regular basis, and reaching the target popula-tion
earlier in their pregnancies.
¾ Collecting urine analysis information—If a woman sub-mits
to urine analysis tests through a substance abuse treat-ment
center, CPS, or the courts, the pilot program will obtain
these results on a regular basis. Because monies are now
available, the program will also offer all other clients the op-portunity
to submit voluntarily to urine analysis tests on a
weekly basis.
¾ Gathering client health and well-being information—The
program’s corrective action plan indicates that tools will be
developed to measure quality-of-life issues (including health
and well-being) at intake and at regular updates.
¾ Reaching women early in their pregnancies—The pro-gram
plans on working with other community organizations
Finding II
27
OFFICE OF THE AUDITOR GENERAL
to provide outreach to women who are in an early stage of
their pregnancies.
Action is needed to put necessary changes into effect—Although
the corrective action plan is an acknowledgement that improve-ments
are needed in these areas, program administrators should
do several things to ensure that outcomes are measured and
achieved. First, all service referrals should be updated so that
program staff and formal collaborators know which services a
woman has received. Participants’ drug use should be moni-tored
through urine analysis, as stated in the corrective action
plan. Further, the program should develop policies and proce-dures
for obtaining urine analysis information from CPS, the
courts, or substance abuse treatment centers on a regular basis.
Procedures should also be established for monitoring client
health and well-being regularly through follow-up assessments.
Finally, the pilot program needs to establish procedures for
working with community organizations to recruit more women
to participate in the program and during the early stages of their
pregnancies.
Recommendations
1. When referrals are updated by the program coordinator or
collaborators, they should indicate whether or not a client re-ceived
a service.
2. The pilot program should monitor client drug usage through
regular urine analysis testing, and establish policies and pro-cedures
for obtaining this information from CPS, the courts,
or substance abuse treatment centers.
3. The pilot program should ensure that 3-month follow-up in-terviews
are completed for all clients. The follow-up data
should be promptly shared so that collaborators can track cli-ent
progress.
4. The pilot program should develop procedures for working
with community organizations to recruit more women to
participate in the program and earlier in their pregnancies.
28
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
29
OFFICE OF THE AUDITOR GENERAL
STATUTORY EVALUATION COMPONENTS
Pursuant to Laws 1998, Ch. 176, §3, the Office of the Auditor
General is required to include the following information in the
Perinatal Substance Abuse Pilot Program evaluation.
B.1. The number, type, and location of integrated service
models funded under this act.
As described in the Introduction and Background section
(pages 1 through 11), only one integrated service model
was funded in July 1999. The EMSA (Expectant Mothers
with Substance Abuse) Esperanza program is housed at
El Rio Health Center in Tucson, Arizona. The pilot pro-gram
officially ends in June 2002.
B.2. The characteristics of the population included in
each of the integrated service models.
The program enrolls women who are:
¾ Pregnant or up to 1 year postpartum;
¾ AHCCCS enrolled or eligible; and
¾ Using drugs or at risk of using drugs; or
¾ At risk of losing child custody due to drug use.
B.3. The services provided by the collaborative commu-nity
partnerships and the models of collaboration
used for each integrated service model.
Collaborators and the services they provide are described
in the Introduction and Background (see pages 1 through
11). There are eight formal collaborators who have signed
Statutory Evaluation Components
30
OFFICE OF THE AUDITOR GENERAL
memos of agreement to participate in the pilot program.
These collaborators represent medical providers, sub-stance
abuse treatment facilities, a domestic violence
agency, and a homeless drop-in shelter. Some of the ser-vices
provided are as follows:
¾ Medical Facilities—El Rio Obstetrics and Gynecol-ogy
and El Rio Midwifery are participants in the col-laboration.
The providers offer medical services for
women throughout their participation in the pilot
program. The Rural Health Office’s mobile health
clinic provides medical care and outreach for com-munities
and individuals who may not have access to
health care.
¾ Substance Abuse Treatment Facilities—CODAC
Las Amigas and The Haven are residential substance
abuse treatment centers. Women may stay at the cen-ters
for 6 to 12 months.1 They receive substance abuse
counseling and life-skills training, and in some cases,
may have their children living with them on-site. La
Frontera operates a methadone maintenance clinic.
Clients receive daily doses of methadone in addition
to case management and nursing services.
¾ Domestic Violence Agency—The Brewster Center is
a domestic violence facility that offers both advocacy
and shelter. Women receive services ranging from le-gal
assistance to one-on-one counseling at both the
outreach facility and within the shelter.
¾ Homeless Drop-In Shelter—Casa Paloma is a home-less
drop-in shelter. Women receive basic necessities
such as food, and may use the showers or laundry fa-cilities.
The shelter also maintains bed space for
women who may stay in the residence for up to 2
years.
1 Women can stay at The Haven for up to 6 months and at CODAC Las
Amigas for up to 12 months.
Statutory Evaluation Components
31
OFFICE OF THE AUDITOR GENERAL
In addition to the 8 formal collaborators, 24 agencies par-ticipate
in the program as informal collaborators. This
means that they do not have access to the Health Pro sys-tem.
However, they provide services to program clients
and they attend and participate in the monthly collabora-tor
meetings. They represent medical, behavioral health,
and social service providers, as well as a school district,
Native American tribes, and legal counsel.
B.4. General demographic and treatment characteristics
of the population served, including information from
the intake and assessment screening.
Demographic information is reported for 67 clients en-rolled
in the program from November 1999 to May 2001.
The typical pilot program client has given birth to two
children, is Hispanic, 29 years old, not married, unem-ployed,
and has less than a high school education.
¾ Children—On average, pilot program clients have
given birth to two children. The number of children
ranges from zero (currently pregnant) to six.
¾ Ethnicity—The majority of the women (46 percent)
are Hispanic, 30 percent are Caucasian, 18 percent are
Native-American, and 6 percent are African-
American.
¾ Age—Women enrolled in the pilot program are, on
average, 29 years old, and range in age from 19 to 42.
¾ Marital Status—80 percent of the clients are not mar-ried;
15 percent are married; and the remaining
women (5 percent) are separated from their spouse.
¾ Income and Employment Status—Pilot program
clients have an average monthly income of $322, rang-ing
from $0-$4,000 per month. Approximately 78 per-cent
of the clients are unemployed. For those women
who are unemployed, 34 percent report no income,
and the primary source of income for another 43 per-
Statutory Evaluation Components
32
OFFICE OF THE AUDITOR GENERAL
cent is government assistance, including food stamps
or temporary assistance for needy families (TANF).
The remaining 22 percent of the women work on a
full- or part-time basis.
¾ Education—56 percent of the women have less than
a complete high school education and 44 percent have
a high school education, equivalent, or higher. Only 7
percent of the clients are currently attending school.
¾ Drug Use History—Approximately 81 percent of the
clients report using, at the time of intake, multiple
drugs and another 18 percent use only one type of
drug. The remaining 1 percent (one client) reported
not using any drug. The three most commonly used
drugs are cocaine, alcohol, and marijuana. Over half
of the clients (61 percent) have used one or more types
of drugs before intake. A typical client first used drugs
between the ages of 17 and 20. Finally, 51 percent of
the women report having received prior treatment for
alcohol or drug abuse, which could include detoxifica-tion
or outpatient treatment.
B.5. General information on the short-term and long-term
outcomes of the services provided, including:
¾ Successful strategies for reducing or eliminating
substance-abusing behaviors—The program did
not collect sufficient data needed to assess outcomes
(see Finding II, pages 21 through 27). The program
did not sufficiently monitor client drug use, so evalua-tors
were unable to determine if any of the clients ac-tually
reduced or eliminated substance-abusing be-haviors.
In addition, the program did not consistently
use the Health Pro system to provide information
about the services the clients received.
¾ The status of the woman’s and the family’s well-being,
including general health, employment, and
housing status—Again, the program did not ade-quately
monitor any improvements to a client’s health
or well-being. Although these traits were assessed at
Statutory Evaluation Components
33
OFFICE OF THE AUDITOR GENERAL
program intake, they were not consistently monitored
at the required three-month intervals.
¾ The drug status of the infant at birth—As discussed
in Finding II (pages 21 through 27), infant birth out-comes
cannot be attributed to a woman’s participa-tion
in the pilot program because too few women en-tered
the program at each stage of pregnancy. How-ever,
information is available on the health and legal
status of 43 children born to program clients.
  Drug Toxicity—Of the 43 newborns, 22 tested
positive for drugs at birth.
  Birth Weight—The average weight of babies born
to program clients is 6 pounds, 6 ounces. Accord-ing
to the National Healthy Start Association, a
birth weight under 5 pounds, 8 ounces is consid-ered
to be a low birth weight. Evaluators com-pared
the birth weights of babies born with posi-tive
drug toxicology screens and those with nega-tive
toxicology screens and found the weights to
be 6 pounds, 3 ounces, and 6 pounds, 10 ounces,
respectively.
  Apgar Score—The pro-gram
infants have an av-erage
5-minute Apgar
score of 8.5, with scores
ranging from 0 to 10. The
Apgar score is a tool
used immediately after
birth to evaluate a child’s
condition. A baby is
rated with a score of zero
to two for each of the five
qualities, with two being the best condition. An
overall score of seven or higher indicates that the
baby is in good condition.
  Legal Custody— Child Protective Services (CPS)
has legal custody of 19 of the 43 children.
Apgar—The five
qualities monitored at
1, 5, and 10 minutes
after birth:
Appearance (color)
Pulse (heartbeat)
Grimace (reflex)
Activity (muscle tone)
Respiration (breathing)
Statutory Evaluation Components
34
OFFICE OF THE AUDITOR GENERAL
¾ The average length of treatment and average
costs compared with estimated costs of non-treatment—
Because the pilot program did not collect
sufficient information about the services clients re-ceived,
evaluators cannot compare the costs of treat-ment
for participating women against costs for
women who did not participate. Further, since the pi-lot
program does not directly provide services, it does
not have information on the costs of the services that
participants may receive.
¾ The number of months the substance-abusing
woman achieves a drug- and alcohol-free status—
The number of months a woman achieves a drug-and
alcohol-free status is unknown because the pro-gram
failed to conduct regular followups with clients
(see Finding II, pages 21 through 27). As of March
2001, none of the clients had received the correct
number of followups, which is how the program
agreed to collect self-reported drug use data.
¾ The relapse rates for women who return to sub-stance-
abusing behaviors after achieving drug-and
alcohol-free status—Relapse rates are also not
reported due to the absence of critical drug use infor-mation
(see the above paragraph and Finding II,
pages 21 through 27).
B.6. Pursuant to Laws 2000, Ch. 393, §13, the Office of the
Auditor General is to include a report on the ex-panded
services and additional populations served
with the $200,000 appropriation of TANF funds.
This portion of the evaluation could not be completed as
required. The program’s plan for using the funds was not
approved until March 2001; thus, it was unable to spend
any of the money until that point.
OFFICE OF THE AUDITOR GENERAL
Agency Response
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
Leadership for a Healthy Arizona
Office of the Director
1740 W. Adams Street JANE DEE HULL, GOVERNOR
Phoenix, Arizona 85007-2670 CATHERINE R. EDEN, DIRECTOR
(602) 542-1025
(602) 542-1062 FAX
Ms. Debra K. Davenport
Auditor General
Office of the Auditor General
2910 North 44th Street, Suite 410
Phoenix, Arizona 85004
Dear Ms. Davenport:
Thank you for giving us an opportunity to respond to your office's evaluation of the Perinatal
Substance Abuse Pilot Program. We agree with the report, both of its findings, and all of its
recommendations. We plan to implement both findings' stated recommendations, should the
Legislature choose to continue the program.
We are particularly pleased that you and your staff highlighted the many accomplishments of this
program. Thirty-two agencies successfully collaborated in providing integrated medical and
behavioral treatment and social services to pregnant and post-partum women. Collaborators met
regularly, learned about services available to these women, and reported high satisfaction with the
pilot project. The bottom line is that 58 percent of the babies born to mothers who enrolled in the
program before giving birth were born drug- free. Moreover, 85 percent of these babies were born
with a normal birth weight. Given the tremendous social and financial ramifications of babies being
born drug addicted or with low birth weight, we believe theses numbers indicate that the program was
ultimately a success.
We regret that the Auditor General was not able to conclude definitely whether the program had a
positive outcome due to data limitations. We are working to improve our data collection efforts, as
recommended in your report. The recent addition of TANF monies to the program now allows us to
purchase the urinalyses testing recommended in the report. Indeed, we recently entered into a
contract to purchase such services. This will enhance our ability to monitor our clients and
demonstrate positive program outcomes. Other data collection obstacles, such as enticing volunteer
collaborators to enter data into a database, will be addressed as the program matures.
Leadership for a Healthy Arizona
In your report, you note that the health status of newborns could not be attributed to the program
because most women entered the program late in their pregnancy or postpartum. You recommend that
the program work to recruit more participants earlier in their pregnancies. While we will, as
recommended in the report, place added effort into reaching program participants earlier in their
pregnancies, we believe it is necessary to recognize the difficulties in doing so, and the health and
social benefits of treating these women at any stage in their pregnancies and postpartum.
Thank you for giving us this opportunity to respond to the report. We appreciate your staff's
professionalism and responsiveness in conducting this evaluation.
Sincerely,
Catherine R. Eden
Director
Other Performance Audit Reports Issued Within
the Last 12 Months
01-10
Future Performance Audit Report
Homeless Youth Intervention Program
01-1 Department of Economic Security—
Child Support Enforcement
01-2 Department of Economic Security—
Healthy Families Program
01-3 Arizona Department of Public
Safety—Drug Abuse Resistance
Education (D.A.R.E.) Program
01-4 Arizona Department of
Corrections—Human Resources
Management
01-5 Arizona Department of Public
Safety—Telecommunications
Bureau
01-6 Board of Osteopathic Examiners in
Medicine and Surgery
01-7 Arizona Department
of Corrections—Support Services
01-8 Arizona Game and Fish Commission
and Department—Wildlife
Management Program
01-9 Arizona Game and Fish
Commission—Heritage Fund
01-10 Department of Public Safety—
Licensing Bureau
01-11 Arizona Commission on the Arts
01-12 Board of Chiropractic Examiners
01-13 Arizona Department of
Corrections—Private Prisons
01-14 Arizona Automobile Theft
Authority
01-15 Department of Real Estate
01-16 Department of Veterans’ Services
Arizona State Veteran Home,
Veterans’ Conservatorship/
Guardianship Program, and
Veterans’ Services Program
01-17 Arizona Board of Dispensing
Opticians
01-18 Arizona Department of Correct-ions—
Administrative Services
and Information Technology
01-19 Arizona Department of Education—
Early Childhood Block Grant
01-20 Department of Public Safety—
Highway Patrol
01-21 Board of Nursing
01-22 Department of Public Safety—
Criminal Investigations Division
01-23 Department of Building and
Fire Safety
01-24 Arizona Veterans’ Service
Advisory Commission
01-25 Department of Corrections—
Arizona Correctional Industries
01-26 Department of Corrections—
Sunset Factors
01-27 Board of Regents
01-28 Department of Public Safety—
Criminal Information Services
Bureau, Access Integrity Unit, and
Fingerprint Identification Bureau
01-29 Department of Public Safety—
Sunset Factors
01-30 Family Builders Program