Performance audit, Department of Health Services' Tobacco Education and Prevention Program

              State of Arizona
Office
of the
Auditor General
PERFORMANCE AUDIT
Report to the Arizona Legislature
By Debra K. Davenport
Acting Auditor General
September 1999
Report No. 99-17
DEPARTMENT
OF
HEALTH SERVICES'
TOBACCO EDUCATION
AND PREVENTION
PROGRAM
The Auditor General is appointed by the Joint Legislative Audit Committee, a bipartisan committee
composed of five senators and five representatives. His mission is to provide independent and impar-tial
information and specific recommendations to improve the operations of state and local government
entities. To this end, he provides financial audits and accounting services to the state and political
subdivisions and performance audits of state agencies and the programs they administer.
The Joint Legislative Audit Committee
Senator Tom Smith, Chairman
Representative Roberta Voss, Vice-Chairman
Senator Keith Bee Representative Robert Burns
Senator Herb Guenther Representative Ken Cheuvront
Senator Darden Hamilton Representative Andy Nichols
Senator Pete Rios Representative Barry Wong
Senator Brenda Burns Representative Jeff Groscost
(ex-officio) (ex-officio)
Audit Staff
Shan Hays—Manager
and Contact Person (602) 553-0333
Miriam Seymore—Audit Senior
Angelica Gonzalez—Staff
Raena Honan—Staff
William Parker—Staff
Copies of the Auditor General’s reports are free.
You may request them by contacting us at:
Office of the Auditor General
2910 N. 44th Street, Suite 410
Phoenix, AZ 85018
(602) 553-0333
Additionally, many of our reports can be found in electronic format at:
www.auditorgen.state.az.us
2910 NORTH 44th STREET • SUITE 410 • PHOENIX, ARIZONA 85018 • (602) 553-0333 • FAX (602) 553-0051
DEBRA K. DAVENPORT, CPA
ACTING AUDITOR GENERAL
STATE OF ARIZONA
OFFICE OF THE
AUDITOR GENERAL
September 13, 1999
Members of the Legislature
The Honorable Jane Dee Hull, Governor
Dr. James Allen, Director
Department of Health Services
Transmitted herewith is a report of the Auditor General, A Performance Audit of the
Department of Health Services’ Tobacco Education and Prevention Program. This
report is in response to a May 27, 1997, resolution of the Joint Legislative Audit
Committee. This performance audit was conducted as part of the Sunset review set
forth in A.R.S. §§41-2951 through 41-2957. This is the fifth in a series of reports to be
issued on the Department of Health Services.
This report addresses the Program’s need for improved evaluations to measure its
effectiveness in preventing youth from starting to use tobacco, encouraging and
assisting tobacco users to quit, and reducing exposure to secondhand smoke. We
recommend that the Program develop an evaluation strategy and comprehensive plan
that corresponds to Program goals, improve evaluation guidance given to the local
projects it oversees, and coordinate with other entities to obtain useful tobacco-related
information.
In addition, the Program needs to improve its guidance of statewide and local projects
in their tobacco control efforts. For example, it should develop written policies and
procedures, facilitate communication among its statewide and local projects, and fully
utilize its Tobacco Use and Prevention Advisory Committee. Further, the Program
should monitor local projects’ administrative costs, which will require developing clear
guidelines and establishing appropriate reporting requirements. Improved
communication and coordination, along with the effective use of resources, will become
increasingly important, as the Program’s future funding level represents a decrease of
nearly 46 percent from the current level.
September 13, 1999
Page –2-
As outlined in its response, the Department agrees with all of our findings, and will imple-ment
the recommendations. However, the decision to continue the Tobacco Use and Pre-vention
Advisory Committee rests with the Legislature.
My staff and I will be pleased to discuss or clarify items in the report.
This report will be released to the public on September 14, 1999.
Sincerely,
Debbie Davenport
Acting Auditor General
Enclosure
i
OFFICE OF THE AUDITOR GENERAL
SUMMARY
The Office of the Auditor General has conducted a performance
audit of the Department of Health Services’ Tobacco Education
and Prevention Program (Program). This audit was conducted
pursuant to a May 27, 1997, resolution of the Joint Legislative
Audit Committee, and as part of the Sunset review set forth in
Arizona Revised Statutes §§41-2951 through 41-2957. This is the
fifth in a series of six audits of the Department of Health Services.
The Program promotes a tobacco-free Arizona through work in
three goal areas: preventing youth from starting to use tobacco,
encouraging and assisting tobacco users to quit, and reducing
exposure to secondhand smoke. With a staff of 16, the Program
oversees tobacco control efforts on both local and statewide lev-els.
On the local level, the Program has intergovernmental
agreements with each county health department to provide to-bacco
cessation counseling, conduct youth awareness programs,
and establish community advisory committees. The Program
also has intergovernmental agreements with various entities to
support statewide tobacco control efforts and complement local
activities. Statewide efforts include a tobacco control advertising
campaign, the Arizona Smokers’ Helpline, materials and infor-mation
clearinghouses, training for tobacco cessation counselors,
and evaluations of Program performance.
Improved Evaluations Are Needed
to Measure Program Effectiveness
(See pages 7 through 16)
Evaluations at the state and local levels have not yet produced an
adequate assessment of the Program’s tobacco control efforts.
Arizona Laws 1995, Chapter 275 mandates evaluations to deter-mine
whether the Program’s efforts are cost-effective and having
any impact on tobacco use. Several of the Program’s attempted
evaluations have encountered logistical and design problems.
For example, the Program spent $900,000 on an evaluation that
had to be terminated because it did not meet the Program’s
The Program oversees to-bacco
control efforts on both
local and statewide levels.
Summary
ii
OFFICE OF THE AUDITOR GENERAL
information needs. In addition, the Program has been unable to
establish a baseline on tobacco use among youth, which is an
essential part of knowing the impact of its advertising campaign
and other activities. The Program has had more success in meas-uring
the prevalence of adult tobacco use, and expects to issue its
first follow-up report on this topic by the end of 1999. More re-cent
evaluations, such as a project that will provide a statewide
assessment of cessation services by combining evaluation infor-mation
from local projects, are more promising but have not yet
been completed. Overall, the Program’s evaluation approach to
date leaves it far short of knowing whether its programs are
working.
To improve its ability to evaluate the effect of its statewide and
local activities, the Program needs to develop an evaluation
strategy, improve its guidance to local project staff, and explore
opportunities to coordinate its efforts with other entities that
collect relevant data or conduct other studies of the Program’s
target populations. The Program only recently hired a statewide
evaluator, and as a result now has a better opportunity to make
these improvements. The evaluation strategy should incorporate
specific goals, objectives, and performance indicators, and it
should help identify ways to better collect evaluation data. For
example, since at least three other entities are studying youths’
attitudes and behaviors, some schools are declining to participate
in school-based surveys. As an alternative, the Program can seek
ways of using data collected by others. The strategy should also
link statewide and local project efforts to the Program’s outcome
objectives, in accordance with Centers for Disease Control
guidelines (the federal agency responsible for coordinating to-bacco
control efforts). Finally, in assembling its evaluation strat-egy,
the Program should seek input from local project staff and
incorporate recommendations from a 1998 assessment by the
University of Arizona’s Cancer Center.
Program Needs to Improve Its
Guidance of Tobacco Control Efforts
(See pages 17 through 24)
The Program needs to improve the guidance and communica-tion
it provides to local projects and statewide contractors. The
Program has not developed manuals or written policies and
The Program has yet to
establish a baseline on
youth tobacco use.
Summary
iii
OFFICE OF THE AUDITOR GENERAL
procedures, but instead waits for local project officials to seek
guidance for many day-to-day decisions about their operations.
The guidance provided is often inconsistent or untimely. For
example, local project staff generally have to wait about three
weeks to receive approval to put information about local area
events on the statewide Internet Web site. By then, the events
have already taken place. Similarly, feedback is often delayed for
months on monthly reports submitted by local projects.
The lack of clear and timely guidance and communication has
resulted in duplicated efforts at events, redundant development
of materials, and poor coordination between programs. For ex-ample,
lack of communication between the statewide advertising
program and the Smokers’ Helpline meant that the Helpline was
unprepared to handle a heavy volume of calls when advertise-ments
were broadcast that gave the Helpline’s 800 number.
Likewise, the Program does not consistently inform local projects
about its advertising contractor’s activities. Consequently, both
the advertising contractor and local projects have sent crews to
the same public events.
To effectively guide Arizona’s tobacco control efforts, the Pro-gram
needs to change the level and amount of guidance, making
it more strategic and system-wide. The first step is to create a
strategic plan that would help focus the Program’s efforts and
better coordinate its statewide and local project resources. In the
future, a strategic plan will become increasingly important, be-cause
the Program’s current level of funding will decrease by
nearly half as a buildup of reserves from the Program’s earliest
years is rapidly spent. In developing a strategic plan and refo-cusing
the guidance provided to its contractors, the Program
should build on the efforts it has recently begun in seeking input
fromits contractors and its advisory committee.
Lack of coordination
meant that the Helpline
was unprepared to handle
a heavy volume of calls.
Summary
iv
OFFICE OF THE AUDITOR GENERAL
Program Does Not Monitor
Administrative Costs
(See pages 25 through 28)
Although the Program has established a limit of 10 percent on
administrative costs, it cannot determine how much of the mon-ies
provided for operating local education and prevention proj-ects
are used for administration. It has not provided adequate
guidelines for identifying such costs, and it has not required local
projects to submit expenditure reports that contain enough detail
to show which expenditures are administrative. As a result, the
Program cannot determine the amount or percentage of Pro-gram
monies that are spent on services, such as helping smokers
quit and preventing youth from starting to use tobacco. Since
services are often provided by several layers of entities charging
administrative costs, Program services could account for much
less that 90 percent of local project funding after counties, con-tractors,
and subcontractors each spend up to 10 percent of their
budgets on administrative costs. To ensure that local projects
stay within the 10 percent limit, the Program needs to establish
clear definitions and more complete reporting requirements.
Several layers of entities
charge administrative
costs.
v
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS
Page
Introduction and Background ......................... 1
Finding I: Improved Evaluations Are
Needed to Measure Program
Effectiveness................................................ 7
Evaluation Requirements............................................... 7
Program Effectiveness
Is Not Yet Known........................................................... 8
The Program Can Improve
Statewide and Local Project
Evaluation Efforts ........................................................... 11
Recommendations .......................................................... 16
Finding II: Program Needs To Improve
Its Guidance of
Tobacco Control Efforts .............................. 17
Program Structure .......................................................... 17
Program Does Not Provide Adequate
Guidance or Communication to
Statewide and Local Projects.......................................... 18
Several Factors
Contribute to
Ineffective Guidance....................................................... 21
Formal Planning and
Written Policies Needed
to Enhance Program Efficiency...................................... 23
Recommendations .......................................................... 24
Table of Contents
vi
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS (Concl’d)
Page
Finding III: Program Does Not Monitor
Administrative Costs ................................... 25
The Program Has Not
Adequately Defined or
Monitored Administrative Costs ................................... 25
Program Needs to
Clarify Definitions
and Establish Reporting
Requirements.................................................................. 27
Recommendations .......................................................... 28
Agency Response
Table
Table 1 Arizona Department of Health Services
Tobacco Education and Prevention Program
Statement of Revenues, Expenditures, and
Other Changes in Fund Balance
Years Ended June 30, 1997, 1998, and 1999
(Unaudited)................................................... 4
1
OFFICE OF THE AUDITOR GENERAL
Item 1: Program Goals
n Preventing youth from
starting to use tobacco;
n Encouraging and
assisting tobacco users
to quit; and
n Reducing exposure to
secondhand smoke
INTRODUCTION AND BACKGROUND
The Office of the Auditor General has conducted a performance
audit of the Department of Health Services’ Tobacco Education
and Prevention Program. This audit was conducted pursuant to
a May 27, 1997, resolution of the Joint Legislative Audit Com-mittee,
and as part of the Sunset review set forth in Arizona Re-vised
Statutes §§41-2951 through 41-2957. This is the fifth in a
series of six audits of the Department of Health Services.
Tobacco Education and
Prevention Program
The Tobacco Education and Prevention Program (Program)
originated in a 1994 voter initiative called the Tobacco Tax and
Health Care Act. The Act increased the tobacco sales tax to 58
cents per pack and required that part of the revenue generated
from the increase be used to provide funding for tobacco educa-tion
and prevention. In 1995, to implement the Act, the Legisla-ture
enacted House Bill 2275, which created the Health Educa-tion
Account for tobacco prevention and reduction program and
monies and directed the Department of Health Services to ad-minister
the Account. The bill also established the Tobacco Use
Prevention Advisory Committee, which is required to review
Program activities and make recommendations to the Depart-ment.
The legislation contains a provision that will terminate the
Committee in December 1999.
The Program seeks to promote a
tobacco-free Arizona through a
combination of statewide and local
efforts. These efforts target three
primary goals (see Item 1). To
achieve these goals, the Program has
adopted a comprehensive structure
recommended by the federal
Centers for Disease Control, which
includes the following elements:
Introduction and Background
2
OFFICE OF THE AUDITOR GENERAL
n Local Projects—Intergovernmental agreements with each
county health department support local efforts, such as to-bacco
cessation counseling, youth awareness programs, and
community partnerships established with local not-for-profit
organizations.
n Statewide Advertising Campaign—Under a competitive
contract with an advertising company, the Program supports
a highly visible, statewide tobacco control advertising cam-paign.
The nationally recognized campaign employs a vari-ety
of creative approaches including television and radio
commercials, print advertisements, a merchandise center for
campaign-related materials, and a mobile “Ash Kicker” vehi-cle
with tobacco control displays aimed at youth.
n Statewide Support Services—The Program provides
funding and support for the Arizona Smokers’ Helpline,
three information and materials resource clearinghouses, a
Web site, and other services. These services are provided
through intergovernmental agreements with Arizona’s three
state universities.
n Statewide Cessation and Evaluation Services—The Ari-zona
Cessation Training and Evaluation Program (ACTEV),
a Program-funded partnership between the University of
Arizona and local projects, provides tobacco cessation certifi-cation
and evaluation training for local projects and
healthcare providers. The certification training focuses on
skills required to encourage and assist tobacco users to quit
and remain tobacco-free. The evaluation training teaches the
skills necessary to evaluate local projects’ tobacco cessation
activities.
n Local and Statewide Enforcement Programs—The Pro-gram
funds efforts to ensure compliance with laws restricting
tobacco sales to minors. State and federal monies are allo-cated
for enforcement. The Program receives grant monies
through the federal Food and Drug Administration (FDA)
enforcement program. The federal monies are administered
by the Program and provided to communities for enforce-ment.
Additionally, state-funded local projects in some areas,
such as the City of Mesa, currently include enforcement in
their activities.
Introduction and Background
3
OFFICE OF THE AUDITOR GENERAL
Budget, Personnel,
and Organization
The Program is primarily funded through state tobacco taxes
with some additional funding from federal grants. The Program
receives 23 percent of each dollar derived from the 58-cent-per-pack
Arizona tobacco sales tax. The Program’s share of fiscal
year 1998-99 tobacco tax revenue amounted to approximately
$26.5 million. In addition, the Program has approximately $46
million in available carryover funds and interest earned, since
the Legislature capped Program spending at $10 million and $15
million during its first and second years of operation, respec-tively.
The Legislature later authorized an additional $9 million
in available monies for the Program’s second year. (See Table 1,
page 4.) Currently, the Legislature restricts Program spending to
no more than 95 percent of the previous year’s tobacco tax reve-nues.
For fiscal year 1998-99, the Program anticipates expenditures of
approximately $32.7 million from state monies, including $1.25
million in transfers to the Arizona Department of Education for
tobacco education in the public schools. The Program funds
school-based tobacco control education efforts in accordance
with federal Centers for Disease Control recommendations. Pro-gram
expenditures primarily consist of services provided at the
statewide and local projects level. A state Program office admin-isters
the tobacco education and prevention projects as follows:
The Program office, housed within the Division of Public Health
Services, is authorized 16 staff. The staff are directed by an office
chief and structured into major program areas, including local
projects support, statewide support, financial management,
marketing, and evaluations. Office administration expenditures
for fiscal year 1998-99 were $726,000.
Statewide functions are performed mainly through a contract
with an advertising company and intergovernmental agree-ments
with all three state universities. Expenditures for such
services were approximately $21.4 million in fiscal year 1998-99,
including $13.5 million for the advertising campaign contract.
The Program anticipates
expenditures of approxi-mately
$32.7 million
from state monies.
Introduction and Background
4
OFFICE OF THE AUDITOR GENERAL
Table 1
Arizona Department of Health Services
Tobacco Education and Prevention Program
Statement of Revenues, Expenditures, and Other Changes in Fund Balance
Years Ended June 30, 1997, 1998, and 1999
(Unaudited)
1997 1998 1999
Revenues:
Tobacco taxes $27,251,281 $26,779,642 $26,457,461
Interest on investments 2,145,792 2,745,938 2,777,685
Federal grants and contracts 48,501 81,962 264,964
Other                  2,653
Total revenues 29,445,574 29,607,542 29,502,763
Expenditures:
Personal services 402,929 393,543 461,315
Employee related 81,589 84,280 95,227
Professional and outside services 1 13,181,941 16,250,403 17,032,162
Travel, in-state 18,936 9,703 11,994
Travel, out-of-state 11,262 9,005 11,629
Aid to organizations 2 4,394,847 7,405,239 14,960,702
Other operating 107,725 50,200 112,442
Capital outlay 25,269 37,026 2,354
Allocated costs 10,360 21,074 30,898
Total expenditures 18,234,858 24,260,473 32,718,723
Excess of revenues over(under) expenditures 11,210,716 5,347,069 (3,215,960)
Fund balance, beginning of year 32,437,568 43,648,284 48,995,353
Fund balance, end of year 3 $43,648,284 $48,995,353 $45,779,393
1 Includes $11.5, $13.2, and $13.5 million expended for media and sports contracts in 1997, 1998, and 1999,
respectively.
2 Includes $3.1, $6.1, and $10.6 million passed through to counties for local projects in 1997, 1998, and
1999, respectively. In addition, includes $2.9 million passed through to Arizona universities in 1999 for
statewide functions and $1.25 million passed through to the Arizona Department of Education for to-bacco
education at the public schools in 1998 and 1999.
3 The Department has legislative guidelines for spending its fund balance.
Source: The Arizona Financial Information System Revenues and Expenditures by Fund, Program, Organi-zation,
and Object and Trial Balance by Fund reports; and Department-provided financial infor-mation
for the years ended June 30, 1997, 1998, and 1999.
Introduction and Background
5
OFFICE OF THE AUDITOR GENERAL
Local projects funded by the
Program through intergovernmen-tal
agreements are operated by
counties, tribal governments, and
their subcontractors. These enti-ties
were allocated about $13.7
million in fiscal year 1998-99 to
develop activities aimed at get-ting
communities, coalitions of
concerned citizens, and interested
groups involved in tobacco
control goals and efforts.
Maricopa County administers its local projects’ activities through
subcontracts for programs and services while the remaining 14
counties directly administer the projects.
Audit Scope and Methodology
A combination of methods was used to study the issues ad-dressed
in this audit. For example, Auditor General staff:
n Reviewed the minutes from a sample of 8 Tobacco Use Pre-vention
Advisory Committee meetings held from 1995 to
1998 and interviewed 7 of the 10 Committee members.
n Reviewed contracts, intergovernmental agreements, and
evaluations for statewide and local projects. Interviews were
conducted with 15 members of local projects staff represent-ing
all 15 Arizona counties, in addition to 7 local projects’
subcontractors.
n Attended meetings of the Coalition for a Tobacco Free Ari-zona,
the Program’s 4th annual conference, and regional
meetings with the local projects held by the Program.
n Interviewed representatives from all the organizations pro-viding
statewide support services including the advertising
campaign contractor, the Arizona Tobacco Information Net-work,
the Arizona Prevention Resource Center, and the Ari-zona
Cessation Training and Evaluation Program.
Item 2: Tobacco Settlement
In the future, Arizona expects
to receive substantial funding
from the multi-state tobacco
industry settlement. However,
the use of such funds has not
been determined by the Gov-ernor
and the Legislature. The
future impact of settlement
dollars on the Program fund-ing
is unknown.
Introduction and Background
6
OFFICE OF THE AUDITOR GENERAL
n Conducted an extensive review of literature, including jour-nal
articles, books, Internet Web sites, and reports from other
states.
n Interviewed administrators of similar programs in 7 other
states.1
This audit contains findings and recommendations in three ar-eas:
n The need to improve the Program’s ability to evaluate the
success of statewide and local projects’ tobacco control ef-forts;
n The need to improve the Program’s oversight of local proj-ects’
efforts; and
n The Program’s inability to monitor local project administra-tive
costs.
This audit was conducted in accordance with government
auditing standards.
The Auditor General and staff express appreciation to the Di-rector
of the Department of Health Services, the Assistant Direc-tor
for the Division of Public Health Services, the management
and staff of the Tobacco Education and Prevention Program, the
management and staff of contracted statewide support services,
and local project staff for their cooperation and assistance
throughout the audit.
1 Other states contacted were California, Florida, Massachusetts, Min-nesota,
Mississippi, Oregon, and Texas. These states were selected be-cause
they had funding sources and goals similar to Arizona’s.
7
OFFICE OF THE AUDITOR GENERAL
FINDING I IMPROVED EVALUATIONS ARE
NEEDED TO MEASURE
PROGRAM EFFECTIVENESS
The Program has not adequately assessed the effectiveness of its
tobacco prevention and cessation efforts. Improvements in
statewide and local evaluations would result in better informa-tion
about the Program’s success. The Program’s newness, to-gether
with delays in hiring an evaluator to coordinate the vari-ous
evaluation efforts, contributed to the slow start. An evaluator
is now in place, giving the Program the opportunity to address
past problems. To maximize the value of evaluation efforts, the
Program should work with national experts and local projects to
develop an evaluation strategy, including a plan to provide out-come
measures and other data useful to state and local project
staff.
Evaluation Requirements
Laws 1995, Chapter 275 requires biannual program evaluations
to determine whether the Program’s education, prevention, and
advertising campaigns are cost-effective and having an impact
on tobacco use. Program evaluations should include the results
of all monitoring of tobacco-related behaviors, attitudes, and
health outcomes assessed at regular intervals. In following the
Centers for Disease Control’s tobacco control program model,
Arizona’s Program requires evaluations on both the statewide
and local project levels. As part of their intergovernmental
agreements with the Program, local projects are required to assist
with data collection, including monthly and quarterly reports
that may be monitored and accumulated for ongoing program
evaluation. Local projects are also required to specify the evalua-tion
budget and staffing they will use to conduct Program
evaluations.
Finding I
8
OFFICE OF THE AUDITOR GENERAL
Program Effectiveness
Is Not Yet Known
While the Program has made some efforts to measure the suc-cess
of statewide efforts over the last three years, its effectiveness
has not been determined. Evaluation efforts since the Program’s
1995 inception have yielded inconclusive results; however, the
Program is currently working on several evaluations that may
provide more useful information in the future.
Evaluation efforts to date have not determined effectiveness—
The Program has attempted several evaluations, including a proc-ess
evaluation that was expected to track both its short-term and
long-term progress; an evaluation aimed at assessing the effec-tiveness
of the statewide advertising campaign; and a survey
intended to measure changes in tobacco use. However, logistical
and design problems hampered the success of two of these ef-forts,
and the third has not yet produced meaningful results.
n Process Evaluation—The Program attempted a process
evaluation of local project activities, but discontinued the ef-fort
when the evaluation failed to provide the information
needed. In 1996, the Program contracted with an organiza-tion
for an evaluation it called Open Systems. The Open
Systems effort was designed to examine the quantity and
quality of activities performed by the local projects, and
their progress toward Program goals. However, the evalua-tion’s
design did not incorporate centralized planning. As a
result, the design did not allow it to compare local projects
to one another or to compare their effectiveness over time,
and did not use random sampling or other methods in-tended
to ensure valid results. Further, local project person-nel
reported many problems with the evaluation, including
the amount of time required by the process and frequent
changes in the way they were instructed to implement the
evaluation. Overall, the Program expended approximately
$900,000 on a process evaluation that did not achieve its
goals.
n Youth Tobacco Survey—The Program did not obtain in-formation
on youth tobacco use before starting its advertising
campaign, and it struggled in subsequent efforts to measure
youth’s use of and exposure to tobacco. The Arizona De-
The Program expended
approximately $900,000
before discontinuing a
process evaluation that
did not achieve its goals.
Finding I
9
OFFICE OF THE AUDITOR GENERAL
partment of Health Services’ Bureau of Public Statistics con-ducted
a statewide telephone survey on youth smoking
during 1996 and 1997, after the advertising campaign aimed
at youth had begun. The survey attempted to estimate the
prevalence of tobacco use, identify knowledge about access
to tobacco products, and identify exposure to secondhand
smoke. However, the survey’s timing undermines its poten-tial
usefulness in measuring the Program’s impact, since the
advertising campaign had already begun. In addition, the
survey may have underestimated the prevalence rate of
youth smoking, because it required parental consent and the
youths may not have given honest answers in the presence of
their parents.
In order to determine the Program’s impact, a survey must
be repeated to measure whether prevalence has changed
from the first time the survey was administered. However,
the Program is replacing the existing survey with another
survey using a different methodology. While this will further
delay the Program’s ability to measure its effectiveness, the
new survey should produce more accurate information on
youth tobacco use.
n Advertising Evaluation—An evaluation of the statewide
advertising campaign may yield more useful results, but
key components will not be completed until September
1999. The first component of this advertising evaluation
included a historical report to the Legislature, completed
in October 1998, and a study of the cost-effectiveness of
the advertising campaign, completed in January 1999.
The October 1998 report incorporated a random sample
telephone survey to measure exposure to and recognition
of the 1996, 1997, and early 1998 advertisements and re-ported
changes in attitudes and behaviors during the
first years of the campaign. However, the January 1999
report could not fully answer questions about the cam-paign’s
cost-effectiveness. According to Program offi-cials,
Arizona was the first state that attempted to meas-ure
cost-effectiveness of its advertising campaign. Using
A.C. Neilsen (television) and Arbitron (radio) data, the
study measured the number of people who saw or heard
television, radio, billboard, and other advertisements.
The study also incorporated results of adult and youth
An evaluation of the
statewide advertising
campaign may yield more
useful results.
The Program still lacks a
baseline on the prevalence
of youth smoking.
Finding I
10
OFFICE OF THE AUDITOR GENERAL
smoking prevalence surveys to determine whether peo-ple
remembered seeing the advertisements. It found, for
example, that the cost per person in the adult population
who saw television advertisements was $.01, and the cost
per adolescent who remembered seeing the television
advertisements was $25.66. However, the study was not
designed to measure whether people had changed their
smoking behaviors or attitudes.
A yet-to-be-completed outcome component may yield more
useful results. In order to look more closely at the link be-tween
the advertising campaign and changes in attitudes
and behaviors, the Program contracted with the Arizona
Cancer Center at the University of Arizona, in 1998, to con-tinue
evaluating the advertising campaign. Once completed,
the evaluation’s outcome component will include six waves
of surveys, including an in-depth survey of youth in
schools, in-depth telephone interviews, and a random tele-phone
sample of Hispanic households with women of
childbearing age. The first wave of the outcome component
was completed in late 1998. The fifth and sixth waves of
surveys are due September 30, 1999, and were not com-pleted
by the end of audit fieldwork. To date, the Program
has expended approximately $265,000 for the advertising
evaluation.
Future evaluations may provide better information—In addition
to the previously mentioned evaluations, the Program is cur-rently
working on other evaluations that may provide more
useful information in the future. However, it is too early to de-termine
if these evaluations will yield desired outcomes.
n Adult Tobacco Survey—The Program has had more suc-cess
in its efforts to measure the prevalence of adult tobacco
use than it has had with youth, although it has not yet
measured any changes in prevalence of use since the Pro-gram’s
inception. The initial survey, conducted in 1996 by
the Department’s Bureau of Public Statistics, yielded infor-mation
on smoking prevalence, individual tobacco use his-tories,
and exposure to secondhand smoke. It also gathered
information on reported tobacco control counseling pro-vided
by health care professionals, and measured aware-ness
of the Program’s tobacco control advertising campaign.
The Program has had more
success in its efforts to
measure adult tobacco use.
The cost-effectiveness
study provided very
limited results.
Finding I
11
OFFICE OF THE AUDITOR GENERAL
The Bureau of Public Statistics plans to repeat the survey
and expects to release a report on the results in September
1999. This will be the first time that the Program can identify
any changes in smoking prevalence since it began.
n Project evaluations—Evaluation of several Program-funded
projects should show whether or not those projects
achieve their goals. For example, the Arizona Cessation
Training and Evaluation Project (ACTEV) is a Program-funded
partnership between the University of Arizona and
local projects to help them set up and evaluate cessation
services for tobacco-dependent community members.
ACTEV will use a standardized questionnaire and provide
evaluation feedback to local projects on the effectiveness of their
cessation activities. ACTEV will also conduct a longitudinal
study to track 500 people over time, a study to examine why
people drop out of cessation activities, and a third study to at-tempt
to gauge the power of brief interventions, using a random
sample of the general population.
Similarly, some other projects will include evaluations.
CHAMPS, a tri-university partnership that trains youth to be
“CHAMPS” by resisting tobacco, includes an evaluation compo-nent.
In this project, participating youth will take tests at the
beginning and end of the program to determine what knowl-edge
they gained and whether they changed their attitudes or
behavior. Finally, a pilot project for providing prevention coun-seling
to Women, Infants, and Children (WIC) participants in
four counties also includes an evaluation component to assess
the intervention and its effect on the target population.
The Program Can Improve
Statewide and Local Project
Evaluation Efforts
While several factors have contributed to the Program’s inability
to measure its effectiveness to date, the Program currently has
the opportunity to address those factors in order to better assess
its progress in the future. The Program recently addressed a
Finding I
12
OFFICE OF THE AUDITOR GENERAL
major deficiency in its evaluation structure by hiring a statewide
evaluator. With the new evaluator in place, the Program should
be able to develop an overall evaluation strategy and provide
better direction and technical assistance to local projects. The
Program should also explore opportunities to gain efficiencies by
working with other entities to obtain needed information.
Lack of evaluator, Program’s newness contributed to past
evaluation problems—Delays in hiring a statewide evaluator
contributed to the lack of continuity and focus for the Program’s
evaluations. Until April 1999, the Program operated without a
statewide evaluator. With the evaluator position now filled, Pro-gram
officials anticipate improvements in both statewide and
local project evaluations.
Because officials lacked evaluation expertise and because na-tional
state-operated tobacco control efforts were new, the Pro-gram
made early mistakes that contributed to the difficulties
encountered in evaluations to date. For example, eagerness to
implement key parts of the Program, such as the advertising
campaign, led directly to the Program’s inability to establish
baseline information describing the prevalence of tobacco use
among the campaign’s target populations before the advertise-ments
began to appear. Similarly, inexperience with evaluation,
and a lack of national models to learn from, made it difficult for
management to predict that the process evaluation would not
meet its needs.
The Program should focus and refine its evaluation strategy—
As a first step in improving its approach to evaluations, the Pro-gram
should develop an evaluation plan to guide statewide and
local efforts. Currently, the Program’s evaluation efforts lack
continuity and focus, because no evaluation plan exists to help
guide statewide or local project staff. A strategic plan is necessary
to provide specific goals, objectives, and performance indicators,
and it can also identify ways to better collect data. The new
statewide evaluator intends to develop an evaluation plan by the
end of 1999. In doing so, the evaluator should also consider ob-taining
input from statewide, local project personnel, and other
stakeholders in determining the future of Program evaluations.
In developing this plan, the Program should consider the rec-ommendations
made in a 1998 needs assessment conducted by
The Program should
develop an evaluation
plan to guide statewide
and local efforts.
Finding I
13
OFFICE OF THE AUDITOR GENERAL
the University of Arizona Cancer Center under a contract with
the Program. This $25,000 study, specifically intended to identify
ways to improve evaluations, created a profile of local projects’
current evaluation efforts and made numerous specific recom-mendations.
However, the Program has not implemented the
study’s recommendations for improving its overall evaluation
strategy, such as forming a regional evaluation team, changing
process evaluations, and improving local project efforts. Other
study recommendations pertained to the kinds of evaluations
the Program should conduct in the future, such as another proc-ess
evaluation.
The Program should improve its evaluation guidance to local
projects—At the same time it improves its statewide evaluation
plan, the Program needs to improve its oversight of local project
evaluation efforts. Although local projects must include an
evaluation component in their programs, the Program has not
effectively communicated its evaluation expectations to local
projects. Also, to maximize the usefulness of future local project
evaluation efforts, the Program should link local project per-formance
indicators to statewide goals.
The Program needs to better communicate its evaluation expec-tations
to local projects, as well as improve coordination between
local project and statewide evaluation efforts. According to local
project coordinators and evaluators, communication between the
Program and local projects is inadequate. Local projects repeat-edly
stated that the Program has not provided adequate evalua-tion
guidance or technical assistance. Further, the Program does
not have standardized evaluation practices that could allow local
projects to work together or compare themselves to each other.
Local projects want the new Program evaluator to provide more
evaluation guidance and technical assistance by holding state-wide
evaluation meetings and possibly creating an evaluation
manual that would outline evaluation activities. Local projects
also want the evaluator to provide feedback on whether their
efforts are meeting statewide expectations and contributing to a
reduction in tobacco use.
According to the Centers for Disease Control, the federal agency
that provides guidance to states in implementing a comprehen-sive
approach to preventing and controlling tobacco use, pro-gram
evaluation efforts should build upon and complement
Communication between
the Program and local
projects is inadequate.
Finding I
14
OFFICE OF THE AUDITOR GENERAL
tobacco-related evaluation systems by linking statewide and
local project efforts to outcome objectives. Currently, the Pro-gram
has no structure in place that links performance indicators
to statewide objectives.
The Program could benefit from external opportunities and in-formation—
Finally, after focusing its evaluation strategy and
providing evaluation guidance, the Program can improve its
efforts by working more collaboratively with related entities and
developing and collecting better data. While the Program has
attempted to coordinate evaluation efforts in the past, many
opportunities to improve efficiencies still exist. Specifically, the
Program can improve its evaluation efforts by working with
other research entities, coordinating the administration of school-based
surveys, and gathering available information about to-bacco.
n Coordination with other entities—The Program could col-laborate
with other entities, such as the Disease Control Re-search
Commission, to obtain additional research related to
tobacco. The Commission is a state agency whose purpose is
to protect public health and safety by contracting with indi-viduals
and organizations to conduct research into the cause
and prevention of diseases, including behavioral changes
such as smoking cessation. Currently, the Program’s Office
Chief represents the Director of the Department of Health
Services on the Commission. Because Commission funding
derives from tobacco sales tax revenues, all Commission
contractors are researching tobacco-related diseases in Ari-zona.
Both the Program and the Commission could poten-tially
benefit from collaborating on evaluations. For example,
Massachusetts, another state with a program like Arizona’s,
benefits from such research by contracting out small research
projects to universities and other community researchers.
Approximately five research projects per year are contracted
out to fill in gaps created by other surveys or to follow up on
problems or issues identified in certain populations. Recent
projects include analyses of special population groups, in-cluding
immigrants and elderly minorities.
n Coordination on school-based surveys—According to
local project coordinators and other evaluators, an excessive
number of surveys are conducted in Arizona’s schools. As a
Finding I
15
OFFICE OF THE AUDITOR GENERAL
result, some school districts have declined to participate in
school-based surveys. To reduce this saturation of surveys,
the Program should continue to work with other entities that
already obtain information about youth through school-based
surveys, such as the Arizona Criminal Justice Commis-sion.
The Program is currently coordinating with Behavioral
Health Services on the High Risk Children’s Survey, which
will include tobacco questions.
n Other available information—The Program should work
with other entities to gain a better understanding of tobacco
use in Arizona. Other states’ tobacco programs compile in-formation
gathered by various entities. For example, Oregon,
California, and Massachusetts use their Departments of
Revenue to collect tobacco consumption data from tobacco
sales taxes. In addition, Oregon obtains birth and death in-formation
from their vital statistics office to determine
whether tobacco was used during a pregnancy or whether
tobacco contributed to the death of a person. Oregon also
uses data collected from the Oregon Office of Alcohol and
Drug Abuse Program.
Finding I
16
OFFICE OF THE AUDITOR GENERAL
Recommendations
1. The Program should develop an evaluation strategy, which
includes a comprehensive evaluation plan that corresponds
to Program goals. As part of its evaluation strategy, the Pro-gram
should:
a. Improve planning by seeking the input of national experts and
other state officials regarding ways to establish linkages be-tween
statewide and local efforts;
b. Review and implement recommendations from the University
of Arizona Cancer Center’s Needs Assessment Study to help
refine Program activities;
c. Establish a youth smoking prevalence baseline; and
d. Continue efforts to work with the CDC on implementing the
new Youth Tobacco Survey.
2. The Program should improve its communication with local
projects by:
a. Seeking recommendations for improvement from local project
coordinators and evaluation staff;
b. Providing feedback to local project personnel regarding
evaluation reports and requirements; and
c. Sharing evaluation information gathered by local projects to
help improve local-level evaluations.
3. The Program should attempt to obtain tobacco-related in-formation
from other entities, such as:
a. Tobacco consumption data from the Department of Revenue;
b. Birth and death statistics from the Office of Vital Statistics; and
c. Tobacco-related studies from the Disease Control Research
Commission to help the Program meet its goals.
4. The Program should continue to coordinate surveys with
other entities, particularly existing school-based surveys.
17
OFFICE OF THE AUDITOR GENERAL
FINDING II PROGRAM NEEDS TO IMPROVE
ITS GUIDANCE OF
TOBACCO CONTROL EFFORTS
The Program needs to improve its interactions with local projects
and statewide entities. The Program does not provide adequate
guidance or communication to its statewide and local projects,
which limits its ability to effectively coordinate its resources.
These problems stem from an approach that emphasizes cen-tralized
approval and does not take advantage of general guid-ance
that could be provided by the Program’s advisory commit-tee.
Improvements are needed to increase efficiency, particularly
since much less money will be available once the Program has
spent the surplus funds that accumulated due to spending re-strictions
during its first two years of operation.
Program Structure
The Tobacco Education and Prevention Program oversees and
coordinates Arizona’s tobacco control campaign through local
and statewide efforts. To operate tobacco control local projects,
the Program has intergovernmental agreements with all Arizona
county health departments. These local projects have four duties:
n develop a local coalition of individuals and groups and write
a plan to support the Program goals;
n conduct prevention programs;
n provide cessation services for tobacco users; and
n promote tobacco-free schools, homes, and workplaces.
Local projects vary in their sophistication. Since some local proj-ects
began their efforts at different times, they differ in the num-ber
and levels of services provided. For example, one of the first
local projects, Pima County, began operations in 1996 and has
staff and contractors working in all four Program areas. In
Finding II
18
OFFICE OF THE AUDITOR GENERAL
contrast, the La Paz local project was initiated in 1998 and has
just begun to initiate tobacco control activities after spending its
first year engaged in planning. In addition, during the first three
years of operation, the Director of the Department of Health
Services restricted Program efforts to prevention programs for
youth and cessation services for pregnant women and their
partners. However, some communities used other resources and
initiated activities outside these restrictions.
In addition to local project efforts, the Program also oversees
contractors who conduct statewide tobacco control efforts.
Statewide programs include an tobacco control advertising cam-paign,
an tobacco control literature and materials clearinghouse,
the Smokers’ Helpline, and cessation specialist training. State-wide
efforts are intended to complement local project activities.
.
Program Does Not Provide Adequate
Guidance or Communication to
Statewide and Local Projects
The Program has not adequately guided the tobacco control
efforts of its statewide and local projects. While the Program has
been in operation for four years, local projects lack basic operat-ing
guidelines and timely assistance. Further, the lack of Pro-gram
guidance and communication hinders the effective coordi-nation
of resources among local projects and statewide entities.
The Program does not provide needed guidance or adequate
communication to local projects—The Program has not pro-vided
consistent and timely technical assistance to statewide and
local projects. Because tobacco education and prevention pro-grams
are new, local projects rely on Program staff to provide
guidance and clarify expectations. However, the Program has
not met local projects’ needs with timely information and feed-back.
Specific problems include:
n No clear policies explaining expectations—The Program
has not written policies and procedures or developed a man-ual
for local projects to follow. As a result, local projects have
received inconsistent instructions regarding Program opera-tions.
For example, when the Program directed local projects
to develop cessation programs for tobacco users who wished
Local projects lack basic
operating guidelines and
timely assistance.
Finding II
19
OFFICE OF THE AUDITOR GENERAL
to quit, it had no established guidance for local projects to use
in setting certification standards for cessation specialists. As
part of the Arizona Cessation Training and Evaluation Proj-ect,
a statewide effort to train cessation specialists was just
getting underway. Local projects received changing and con-flicting
instructions until certification standards were written
several months after the training program began.
n Untimely answers and approvals—The Program has been
slow to respond to local projects’ requests for information
and assistance. For example, local project staff cannot submit
updated public information about local area events and
community services through the statewide Internet Web site
without Program office approval. However, approval rou-tinely
takes three weeks, by which time the information is out
of date. Similarly, it took the Program six months to approve
an Internet mailing list and discussion forum that would help
local projects communicate with each other.
n Lack of feedback—The Program has not provided adequate
feedback to its local projects. One valuable resource for feed-back
is the monthly reporting from local projects submitted
for Program staff review and comment. However, the Pro-gram
usually has at least a six-month backlog of reports
waiting for review, before feedback can be given to local
projects. Most guidance given to local projects is conducted
informally through e-mail and verbal updates that Program
staff deliver at regional meetings, unsupported by written
policies or guidelines.
Lack of guidance and communication hinders effective coordina-tion
of resources—The lack of Program guidance and communi-cation
results in an ineffective use of resources. Problems created
include duplication in staffing and materials; incomplete coordi-nation
of tobacco control efforts with schools; and a lack of coor-dination
among the Program’s advertising contractor, other
statewide contractors, and local projects.
n Duplicate staffing—The Program does not consistently
inform local projects about its advertising contractor’s activi-ties.
Consequently, both the advertising contractor and local
projects have sent crews to the same public events.
The lack of written poli-cies
and procedures has
led to inconsistent deci-sions.
Finding II
20
OFFICE OF THE AUDITOR GENERAL
n Duplicate materials—Both state and local projects are dupli-cating
each other’s efforts in creating similar tobacco control
materials. Tobacco prevention and education programs can-not
function without accurate and professional public infor-mation.
Although the Program contracts with Arizona State
University to operate a statewide clearinghouse to create
materials, some local projects are unaware of these efforts.
The statewide clearinghouse is currently creating “quit kits”
and three designs for decals that announce a smoke-free en-vironment.
Since local projects are not regularly involved
with the clearinghouse, they are creating their own “quit
kits” and smoke-free decals. If the Program involved local
projects in such clearinghouse activities, the local projects
would not only be aware of clearinghouse efforts, they could
participate in the development of materials suited to local
needs.
n The Program does not facilitate use of its monies in
schools—The Program has not facilitated communication
with schools regarding the use of tobacco control education
monies. Specifically, the Program has not enforced the provi-sions
of its interagency agreement with the Arizona Depart-ment
of Education, which requires performance standards,
monitoring, and regular reports on the use of Program to-bacco
control monies provided to schools. Although school
settings are crucial for tobacco use prevention, some schools
are unaware that some of the monies they receive from ADE
are Program monies to pay for tobacco control efforts. In ad-dition,
the Program and ADE do not require schools receiv-ing
these monies to work with statewide and local projects.
As a result, schools have purchased curricula or attempted to
develop it themselves, unaware they can obtain it without
cost through the Program’s statewide clearinghouse.
n The Program does not adequately coordinate statewide
and local project activities—Crucial information is not
shared between statewide programs, or between these pro-grams
and local projects, making it more difficult for state-wide
and local project staff to carry out their duties effec-tively.
Experiences with the Smoker’s Helpline, a statewide
program, provide an illustration of these problems. One set
of problems involved the lack of coordination between
Helpline and another statewide program, the advertising
State and local projects
are duplicating each
other’s efforts.
Crucial information is
not shared.
Finding II
21
OFFICE OF THE AUDITOR GENERAL
campaign. The Helpline is staffed by counselors who fill re-quests
for written material about tobacco use and advise
smokers who wish to quit. The Helpline has not been notified
in advance when its toll-free number is broadcast on radio
and television stations throughout Arizona. Such notification
would give Helpline managers an opportunity to prepare for
the additional calls such announcements can bring. On sev-eral
occasions after such announcements, the Helpline re-ceived
hundreds of calls without enough counselors to han-dle
them. Likewise, the Helpline has not been notified when
Spanish-speaking commercials are aired, giving the staff no
opportunity to add enough bilingual counselors. A second
set of problems involved coordination between Helpline and
local programs. The Program has not permitted the Helpline
to refer callers to the local projects in their communities be-cause
the Program has not decided if, or how, referrals
should be made. As a result, callers may not be aware of Pro-gram
services provided by their local project.
Several Factors
Contribute to
Ineffective Guidance
Two main factors have limited the Program’s ability to guide
tobacco control efforts. First, Program management has overem-phasized
supervision of contractors’ day-to-day operations at the
expense of establishing broad policies and creating an effective
network for open communication among its contractors. Second,
the Program has not sought input from its advisory committee,
which could help in developing such policies.
Program has focused on supervising daily operations—Instead
of developing clear guidelines for contractors to follow, the Pro-gram
has required contractors to seek guidance and approval for
all day-to-day operational decisions. Program management con-sidered
such control necessary to establishing a new program,
especially with inexperienced staff working at the local projects
and to ensure that only appropriate activities received funding.
The Program’s supervisory approach may also have contributed
to its failure to obtain advice from local project staff on ways to
improve the Program. Although Program officials occasionally
Finding II
22
OFFICE OF THE AUDITOR GENERAL
asked local project staff to fill out questionnaires, or to voice their
concerns after hearing a presentation on Program decisions
made by the officials, these efforts were inadequate for obtaining
meaningful local project input. Recently, however, the Program
has improved its efforts to communicate and to solicit input. In
regional meetings held early in 1999, local project staff were
asked to comment on a proposed form revision and to provide
their views regarding how the Smokers’ Helpline might refer
callers to the local projects.
Limited input sought from advisory committee—The Program
has not sought input from its advisory committee, which could
assist it in developing broad policies to guide local projects and
statewide contractors in their activities. The Tobacco Use Pre-vention
Advisory Committee (TUPAC) is an advisory group of
physicians, community leaders, and policy makers created by
1995 legislation. TUPAC is chaired by the director of the De-partment
but has no decision-making authority. The Program
does not consult them prior to making decisions, but TUPAC
members are given information and may ask questions. During
the audit, five of the public members of TUPAC were inter-viewed.
Most report they do not interact with local projects and
are unfamiliar with them. In addition, the majority of TUPAC’s
public members expressed confusion over their role and some
are reluctant to continue as members because they do not feel
effective.
The Legislature has recognized the need for stronger external
oversight. In May 1999, the Legislature passed House Bill 2481.
As part of this Department of Health Services’ omnibus bill,
TUPAC would have been replaced by a Tobacco Revenue Use
Spending and Tracking Commission. This legislation would
have given the Commission many of the Program’s current
oversight functions, such as developing a five-year plan for the
use of monies in the Program’s Health Education Account, per-forming
evaluations and reviews of programs and projects
funded from the Account, and approving expenditures of mon-ies
from the Account. However, this legislation was subse-quently
vetoed by the Governor. Without new legislation,
TUPAC will sunset on December 31, 1999.
The Legislature has
recognized the need for
stronger external over-sight.
Finding II
23
OFFICE OF THE AUDITOR GENERAL
Formal Planning and
Written Policies Needed
to Enhance Program Efficiency
In order to improve, the Program needs to change the level and
amount of guidance it provides to statewide and local projects.
The Program has lacked long-term strategic planning, and has
not developed written policies and procedures. Advance plan-ning
is essential to establish productive working relationships
among statewide and local projects and to effectively coordinate
resources.
The Program needs to create a strategic plan for statewide and
local projects. As noted earlier, lack of understanding of roles,
responsibilities, and goals has created problems among local
projects and statewide entities. A strategic plan could focus ef-forts
of the Program and its statewide and local projects and help
effectively coordinate these resources. As part of a strategic plan
to guide tobacco control efforts and effectively coordinate re-sources,
the Program should determine how to cope with de-creased
availability of monies in the future. Currently, the Pro-gram
is spending a previously unused surplus of approximately
$43.6 million from the Health Education Account. By compari-son,
Program management projects future tobacco tax revenues
will generate approximately $20 million annually. The Program’s
future funding level represents a decrease of nearly 46 percent of
its current spending. (For more information on Program fund-ing,
see Introduction and Background, pages 1 through 6.)
The Program needs to develop written policies and procedures
or manuals. Written guidelines would lessen the problem of
local projects constantly seeking guidance from Program staff
and management. Currently, Program guidelines are contained
in separate documents and locations, including intergovern-mental
agreements, annual renewal applications, and day-to-day
e-mail responses to local project questions. The latter have been
stored in the Program’s staff computers since 1995.
Finding II
24
OFFICE OF THE AUDITOR GENERAL
Recommendations
1. The Program should conduct strategic planning with its ad-visory
committee, statewide entities, and local projects, to in-clude:
a. Program goals and objectives;
b. Allocation of resources; and
c. Coordination of Program entities.
2. The Program should establish clear policies and procedures
for statewide and local projects to define responsibilities,
roles, and restrictions.
3. The Program should aggressively facilitate communication
among statewide entities, local projects, and schools in order
to share information and coordinate resources.
4. The Program should fully use the talents and experiences of
its advisory committee and local projects staff to refine its ac-tivities.
5. The Legislature should consider whether to continue the
Tobacco Use Prevention Advisory Committee (TUPAC).
25
OFFICE OF THE AUDITOR GENERAL
FINDING III PROGRAM DOES NOT MONITOR
ADMINISTRATIVE COSTS
The Program cannot determine how much of the monies pro-vided
for operating local education and prevention projects are
used for administration. While its intergovernmental agreements
restrict the amount of Program monies that can be spent on ad-ministrative
costs, the Program cannot determine whether local
projects adhere to the limit. In order to assess compliance with
the limit and, eventually, measure the amount of local project
funding spent on services, the Program needs to develop clear
guidelines for determining administrative costs and establish
appropriate reporting requirements.
The Program Has Not
Adequately Defined or
Monitored Administrative Costs
The Program has no assurance that administrative costs are
within the limit it has established. The Program has not provided
adequate guidelines for determining what should be considered
as administrative costs and has not required local projects to
submit detailed reports. As a result, the Program cannot deter-mine
what portion of its monies has been spent on Program
services.
Program has established limit on administrative costs—The
Program has established a limit on how much of its monies that
local projects can spend on administrative costs. The Program's
intergovernmental agreements restrict local projects to spending
no more than 10 percent of budgeted monies on such costs. This
is similar to state agencies, which typically spend 8 to 10 percent
of their operating budgets on administrative costs.
Program has not developed adequate guidelines for determining
such costs—Despite setting the limit, the Program has not pro-vided
clear guidelines for measuring administrative costs.
The Program has not
provided adequate guide-lines
for determining
administrative costs.
Finding III
26
OFFICE OF THE AUDITOR GENERAL
Its intergovernmental agreements instruct local projects to follow
the standards and practices of the Department's 1987 accounting
manual. However, this manual does not provide guidelines for
administrative costs. Further, the Program uses the term “indi-rect
costs” interchangeably with “administrative costs,” although
the two terms have slightly different meanings. According to A
Dictionary for Accountants,1 administrative costs are expenses
incurred “in the general direction of an organization as a whole.”
Indirect costs, according to the Department’s manual, include
any costs incurred for a joint purpose and benefiting more than
one program. This mix of terms further adds to the potential con-fusion
about which costs should be included.
Program does not require detailed reporting—Even if the Pro-gram
provided clear guidelines regarding administrative costs, it
could not determine whether local project administrative expen-ditures
exceed the 10 percent limit because it does not require
detailed tracking and reporting of costs. The Program requires
financial reports showing total expenditures in categories such as
personal services (salaries) and employee-related expenses
(benefits), but these categories are too broad to permit Program
staff to determine the amount of administrative costs.
As a result, Program cannot measure service costs—As a result
of its inability to measure administrative costs, the Program can-not
determine the amount or percentage of Program monies that
are spent on services such as helping smokers quit and prevent-ing
youth from starting to use tobacco. There can be several lay-ers
of entities charging administrative costs. Services are often
provided by subcontractors, who may work under contract with
other entities that hold contracts with county local projects. Thus,
program services could account for much less than 90 percent of
local project funding after the entities at each level (counties,
contractors, and subcontractors) spend up to 10 percent of their
budgets on administrative costs.
1 Kohler, Eric L. A Dictionary for Accountants. Prentice-Hall, Inc., 1975.
There can be several
layers of entities charging
administrative costs.
Finding III
27
OFFICE OF THE AUDITOR GENERAL
Program Needs to
Clarify Definitions
and Establish Reporting
Requirements
In order to monitor compliance with the 10 percent administra-tive
cost limit and maximize monies spent on tobacco control
efforts, the Program needs to establish clear guidelines and re-porting
requirements. In providing clearer guidelines, the Pro-gram
also needs to differentiate administrative costs from indi-rect
costs. This could be done as part of a thorough update to the
Department’s accounting manual, which has not been revised
since 1987. In setting more detailed reporting requirements, the
Program needs to ensure that the reports will allow staff to as-sess
whether each local project has kept its administrative costs
within the 10 percent limit.
In the future, once it has established a way to monitor compli-ance
with the administrative cost limit, the Program should plan
to measure the percentage of local project budgets that are used
for program services. This will necessitate establishing proce-dures
for tracking costs at every level down to the subcontractors
who actually provide the prevention or cessation services. Only
after such procedures are in place will the Program and its local
projects be sufficiently accountable for their use of tobacco tax
revenues.
Finding III
28
OFFICE OF THE AUDITOR GENERAL
Recommendations
1. The Department of Health Services and the Program should
clarify the restriction on administrative costs by:
a. Defining administrative costs,
b. Differentiating between administrative costs and indirect costs,
and
c. Preparing guidelines for determining which specific costs
should be included in the administrative cost category.
2. The Program should improve its monitoring of administra-tive
costs by establishing reporting requirements for admin-istrative
costs in its intergovernmental agreements.
3. The Department of Health Services should update its Ac-counting
and Auditing Procedures Manual for ADHS Funded
Programs, including adding the administrative cost definition
and guidelines.
4. Once administrative costs are clearly defined and adminis-trative
cost reporting requirements are in place, the Program
should improve its accountability for its use of tobacco tax
revenues by developing a plan for measuring the amount of
local project funding used for providing program services.
OFFICE OF THE AUDITOR GENERAL
AGENCY RESPONSE
Page 1 of 9
ADHS Response to the Performance Audit on
the Arizona Tobacco Education and Prevention Program (TEPP)
Overview
The Arizona Department of Health Services (ADHS) agrees with the recommendations of the audit
team. Overall, the report accurately reflects the challenges faced by the Arizona Tobacco Education
and Prevention Program (TEPP) staff in establishing and administering a statewide, comprehensive
tobacco control program.
In reviewing the progress and problems with the TEPP in the control of tobacco use, it is useful to
remember that Arizona is only the third state in the nation to establish such a program funded by a
tobacco tax. This state is one of the pioneers in comprehensive tobacco control. As such, we have
helped chart the way for others without the benefit of well-established and proven models on which to
base a complex and sophisticated program attempting to help people make more healthful choices B
whether trying to give adolescents the motivation and tools never to begin tobacco use or providing
adult smokers with the motivation and means to stop smoking successfully. The model for a statewide,
comprehensive tobacco control program is complicated, requiring integration of services among all the
components, including media, local projects, and supporting services (such as the Arizona Smokers=
Helpline, the Arizona Cessation Training and Evaluation Project, and the Arizona Tobacco Information
Network).
At the time the initiative was approved by voters, the tobacco prevention program within the ADHS
consisted of only a half-time position funded primarily through preventive health block grant funds from
the Centers for Disease Control and Prevention (CDC). The TEPP was established in early 1995.
Within six months, an initial tobacco control plan for the state was developed and the key components
of the program that is now TEPP were established. Tasks that were accomplished included hiring
additional staff; the preparation, review and award of Requests for Proposals for media, local projects
and the clearinghouse; and writing administrative rules.
The county health departments have been key partners in developing an effective program response.
Although only a few had any experience with tobacco control programs before the implementation of
the statewide project, all have responded enthusiastically in learning about the needs and developing
programs in new areas such as social marketing and aggressive media campaigns. Not surprisingly,
many of the counties experienced the same type of growing pains that faced the TEPP during these early
years.
Staffing has continued to be a critical issue for TEPP, and has been a factor in addressing many of the
findings cited in the Audit Report. For the first three years, the program operated with only five
professional staff and two support staff. Key staffing positions have been added in the past two years,
including a media liaison responsible for monitoring and administering the media contract, three health
educators, and an evaluation coordinator. With this staff complement, we are addressing the issues
raised in the Findings and Recommendations of this Audit Report.
Page 2 of 9
Finding I - Improved Evaluations are Needed to Measure Program Effectiveness
The TEPP has been candid in its reports to the Tobacco Use Prevention Advisory Committee
(TUPAC) and to other external stakeholders about its deficiencies in evaluation. The TEPP actively
recruited to fill the position of evaluation coordinator for nearly two years before finding a qualified
individual, Pamela Goslar, Ph.D., who joined the TEPP staff in April 1999. Dr. Goslar has been
working on a time line for the development of a statewide evaluation strategy (Recommendation 1).
The first draft of this strategy was distributed and discussed with key stakeholders, including local
projects (Recommendation 1a), on September 1, 1999. Information from this discussion will be
incorporated into a final draft that will be presented to TUPAC in October.
As part of her evaluation planning, Dr. Goslar has identified sources of information on evaluation relative
to tobacco control (including the Needs Assessment Study done by the University of
ArizonaBRecommendation 1b). These include the issues and considerations that must be
incorporated into the strategy, the legislative evaluation requirements and programmatic guidelines for
the program, and a summary of evaluation activities that are already underway.
Also, in conjunction with the strategic planning process for evaluation, TEPP will also be developing its
five-year strategic plan. The program has identified measurable goals and objectives that will serve as
the basis for this planning. The TEPP will seek input from its key stakeholders, including local projects
and TUPAC, as recommended by the audit report.
With regard to measuring youth prevalence, in spite of the limitations in use of the results of the 1997
Statewide Youth Survey, we believe that other sources of data can be used to establish youth
prevalence. The Arizona Criminal Justice Commission High School Substance Abuse Survey, which
has collected information on Arizona youth tobacco use as far back as 1993, can be used to determine
prevalence rates before the implementation of TEPP programs. The TEPP is working with the CDC on
conducting the Youth Tobacco Survey (YTS), which has been developed by the CDC for use in all
states. In order to be able to use some of the data collected by the 1997 Statewide Youth Survey, we
will retain key questions and incorporate them into the YTS. It is TEPP=s intent to administer the YTS
in schools next spring, and alternate years after that to establish a longitudinal measure of youth
prevalence rates. However, because we are changing our data collection instrument and shifting from a
telephone-based survey to a school-based survey, it will be at least three years before we can
reasonably expect to have comparable data.
Finding I Recommendations
1. The Program should develop an evaluation strategy, which includes a comprehensive evaluation
plan that corresponds to Program goals. As part of its evaluation strategy, the Program should:
a. Improve planning by seeking the input of national experts and other state officials
Page 3 of 9
regarding ways to establish linkages between statewide and local efforts;
b. Review and implement recommendations from the University of Arizona Cancer
Center=s Needs Assessment Study to help refine Program activities; and
c. Establish a youth smoking prevalence baseline.
d. Continue efforts to work with the CDC on implementing the new Youth Tobacco
Survey.
The finding of the Auditor General is agreed to and the audit recommendation will be
implemented.
2. The Program should improve its communication with local projects by:
a. Seeking recommendations for improvement from local project coordinators and
evaluation staff;
b. Providing feedback to local project personnel regarding evaluation reports and
requirements; and
c. Sharing evaluation information gathered by local projects to help improve local-level
evaluations.
The finding of the Auditor General is agreed to and the audit recommendation will be
implemented.
3. The Program should attempt to obtain tobacco-related information from other entities, such as:
e. Tobacco consumption data from the Department of Revenue;
f. Birth and death statistics from the Office of Vital Statistics; and
g. Tobacco-related studies from the Disease Control Research Commission to help the
Program meet its goals.
The finding of the Auditor General is agreed to and the audit recommendation will be
implemented.
4. The Program should continue to coordinate surveys with other entities, particularly existing
school-based surveys.
The finding of the Auditor General is agreed to and the audit recommendation will be
implemented.
Finding II - Program Needs to Improve Its Guidance of Tobacco Control Efforts
The TEPP agrees with the audit report recommendation that written policies and procedures for all
contractors are needed. In addition to a comprehensive policies and procedures manual, TEPP is
developing a guidance manual for local projects to use in implementing prevention, education, cessation
Page 4 of 9
services and in working toward tobacco-free environments as required by their contracts. We are
striving in this to establish policies and procedures that both provide sufficient direction and allow local
projects and their coalitions to create services based on individual community needs.
We have given clear guidelines to local projects for school-based programs and for cessation. Local
projects have been directed to utilize the CDC Guidelines for School-based Programs as well as the
Agency for Health Care Policy and Research (AHCPR) Guidelines for Cessation. Although we made
changes to the certification standards provided as part of the Arizona Cessation Training and Evaluation
Project (ACTEV), we made many of those changes to reflect feedback received from early piloting of
training programs.
The TEPP acknowledges that we have had difficulty providing timely response and approval to local
projects. We are committed to reducing the turnaround time. We have hired two health educators to
work on local project administration. Their assignment is to assist in the review of local project reports
and to provide feedback to local projects. We no longer have a six-month backlog of reports awaiting
review, and the health educators have improved monitoring the compliance of local projects with
monthly and quarterly reporting requirements.
Communication between local projects and media contractor
This finding cites examples of mis-communication primarily between local projects and the media
contractor, Riester-Robb. The TEPP has aggressively taken steps to ensure improved communication,
including the following:
$ Filling the Media Coordinator Position: This TEPP staff position is dedicated to facilitating
communication between the Media Contractor (Riester-Robb) and TEPP and monitoring
communication between the Media Contractor and the TEPP funded projects (local projects,
Arizona Tobacco Information Network, Arizona Smokers= Helpline).
$ Designating a Local Projects Liaison within Riester-Robb: In 1997, the TEPP directed Riester-
Robb to designate a staff person to be responsible for providing local projects with updates on
developments related to the Statewide Media Campaign and to facilitate feedback from local
projects on creative development, merchandise center products and event sponsorships. Local
projects have indicated that this liaison person has improved their awareness of what is happening
with the media campaign.
$ Requiring each Local Project to Designate a Media Contact: Each local project has designated a
key media contact, and the projects are aware that it is the responsibility of this person to ensure
that information is shared with the project coordinator and other local project staff. In many cases,
the local project coordinator is the media contact. Other local projects have a designated staff
person or a subcontractor to handle media related issues.
Page 5 of 9
Although the local project coordinators acknowledge that they still have some failures in internal
communication within the local projects, the designation of a single media contact person has
facilitated information being shared within the project.
Similar problems were experienced with coordination of statewide media featuring the Arizona
Smokers= Helpline. The Arizona Smokers= Helpline now receives updates on the media plan, which
outlines when commercials will air. Riester-Robb has been directed to include Arizona Smokers=
Helpline in its media updates, and Arizona Smokers= Helpline staff participate in key media creative and
planning meetings.
Role of the Arizona Tobacco Information Network (ATIN) and its relationship to local
projects
The audit report cites the development of Aduplicate materials@ as an issue. The role and function of the
Arizona Tobacco Information Network (ATIN) is to collect tobacco-related education and prevention
materials and make them available to local projects for their use. Local projects are also encouraged to
develop their own Abrief intervention@ materials (such as brochures, handouts, fliers) as appropriate to
address the specific needs of their communities. The ATIN does collect materials that are developed
by individual local projects and makes them available to other projects that may find them appropriate
for their use. The TEPP is encouraging ATIN to share only those project materials demonstrated to be
effective and popular.
To facilitate local project input, the ATIN conducts an annual needs assessment of local projects. This
assessment allows the projects to focus their efforts in the upcoming contract year to suit their needs. In
June 1999, ATIN presented TEPP with the results of a local projects customer-satisfaction survey that
will serve as the basis for ongoing changes and improvements in ATIN services. As with the media
contract, ATIN has a staff position designated to facilitate coordination with local projects.
The TEPP and the ATIN are moving toward a more collaborative approach in the development of
certain materials. One good example of that approach is the Spanish Language Materials Committee
facilitated by ATIN. Local projects were asked to provide ATIN with Spanish language materials they
had developed and/or were using in their communities. A group of staff from local projects was
convened to review those materials and to identify gaps that needed to be filled. After a year=s work,
we have a core of Spanish language materials that can be used by all local projects, including pieces that
were adapted from materials developed by individual local projects. This model for materials
development is being expanded to look at other needs, such as materials for Native Americans, African
Americans and Asian Americans.
In addition to the Spanish Language Materials Committee, ATIN has facilitated a review committee to
look at the materials included in the ATIN catalog for local projects to order in bulk. Local projects are
invited to participate in this activity regularly.
Page 6 of 9
The ATIN has also developed uniform tobacco-free schools signage that will be distributed to local
projects and school districts to assist in the implementation of new tobacco-free schools legislation. As
TEPP moves toward promoting tobacco-free work sites, it will identify signage and materials currently
being used by local projects and adapt them for use by all projects.
Monitoring of TEPP funds by Arizona Department of Education (ADE)
With regard to the use of TEPP funds by the Arizona Department of Education (ADE), TEPP has been
working with the ADE on the use of tobacco education and prevention funding by school districts. Last
year, TEPP directed ADE to dedicate these resources for the development and enforcement of
tobacco-free school policies. There is specific language in the application going to school districts that
directs them to work with local projects as a requirement of receipt of this funding. We have found it
difficult to monitor compliance with these requirements, however.
For the upcoming year, TEPP has again emphasized that these funds should be used for enforcement of
tobacco-free school policies, and with the new legislative restriction prohibiting tobacco on school
campuses and school-related events, districts have an opportunity to take advantage of these funds to
implement the provisions of the law.
Finally, the TEPP has taken steps to improve coordination of information among statewide and local
project contractors. Coordinating communication and activities among the main components (local
projects, ATIN, Arizona Smokers= Helpline, Statewide Media and ACTEV), is challenging, and TEPP
has learned from experience that activities within one component invariably affect the work of the other
three. To that end, we make significant efforts to ensure representation from each of the components
affected by a specific issue in meetings and/or correspondence related to the topic. The TEPP is
moving to the Awork group@ concept which brings these representatives together to discuss specific
activities. The TEPP has also designated specific staff within each of the statewide projects to serve as a
liaison to local projects to facilitate that communication.
The audit report cites TEPP restriction of referral of calls from Arizona Smokers= Helpline to local
project services. In the past, the TEPP did delay referral of callers until several key issues had been
clarified. It was important to establish the ability of local projects to develop localized services, and
more importantly, to handle and track those referrals. Many local projects were not prepared to accept
referrals until this fiscal year, and TEPP has directed that statewide media (primarily television) would
only be tagged with the Arizona Smokers= Helpline number, and that localized print and radio would be
appropriate for local project tagging and referral. There were also logistical issues with Arizona
Smokers= Helpline ability to refer callers efficiently to local projects, or to ensure that the information
they had was accurate and current. The Arizona Smokers= Helpline is now capable of referring callers
to individual local projects, which ensures that callers from outlying areas can be referred to local
services.
Page 7 of 9
Finding II Recommendations
1. The Program should conduct strategic planning with its advisory committee, statewide entities, and
local projects, to include:
a. Program goals and objectives;
b. Allocation of resources; and
c. Coordination of Program entities.
The finding of the Auditor General is agreed to and the audit recommendation will be
implemented.
2. The Program should establish clear policies and procedures for statewide and local projects to
define responsibilities, roles, and restrictions.
The finding of the Auditor General is agreed to and the audit recommendation will be
implemented.
3. The Program should aggressively facilitate communication among statewide entities, local projects,
and schools in order to share information and coordinate resources.
The finding of the Auditor General is agreed to and the audit recommendation will be
implemented.
4. The Program should fully use the talents and experiences of its advisory committee and local
projects staff to refine its activities.
The finding of the Auditor General is agreed to and the audit recommendation will be
implemented.
5. The Legislature should consider whether to continue the Tobacco Use Prevention Advisory
Committee (TUPAC).
The finding of the Auditor General is agreed to and the audit recommendation will be
implemented.
Finding III - Program Does Not Monitor Administrative Costs
The TEPP=s overall direction to its contractors is to ensure that the maximum amount of funding is
dedicated to services, with limited administrative costs. To that end, TEPP has held contractors to a
10% limit for administrative costs. For the contracts beginning July 1, 1999, TEPP has clarified that
policy to state more clearly that contracts are limited to 15% of Personnel and Employee Related
Expenses.
Page 8 of 9
While this limit on administrative costs applies to primary contracts, TEPP does not require, but
recommends that contractors carry that policy forward to their subcontracts. The TEPP also does not
distinguish between administrative costs and indirect costs.
In preparing this response to the audit report (Recommendation 1a through 1c), the Division of
Business and Financial Services (BFS) within ADHS has advised that the ADHS does not have a
definition of administrative or indirect costs. The ADHS has advised the TEPP that it is reviewing this
issue and will provide a definition of administrative and indirect costs and guidance for implementing the
definition with the projects and their subcontractors.
The Accounting and Auditing Procedures Manual for ADHS Funded Programs, is scheduled for a
review (Recommendation 3). In the interim, the TEPP will consult with the ADHS Office of Auditing
and develop broad distinctions between Aadministrative@ and Aservice@ costs, and has asked that they
survey a sample of local project and statewide contracts through the subcontract level to determine if a
disproportionate amount is being spent on administrative costs. This may assist in defining the problem
with regard to the audit finding as well as in arriving at a definition of administrative and indirect costs.
As soon as a definition has been established, TEPP will work with the appropriate representatives from
the county health departments, the universities, other contractors, and representatives from ADHS to
look at how such a definition will be applied to the TEPP contracts.
With those guidelines in place, the TEPP can then proceed to identify an appropriate means for
monitoring these costs among contractors on an ongoing basis, either through their current reporting
requirement or through an independent audit (Recommendations 2 and 4).
Finding III Recommendations
1. The Department of Health Services and the Program should clarify the restriction on administrative
costs by:
a. Defining administrative costs,
b. Differentiating between administrative costs and indirect costs, and
c. Preparing guidelines for determining which specific costs should be included in the administrative
cost category.
The finding of the Auditor General is agreed to and the audit recommendation will be
implemented.
Page 9 of 9
2. The Program should improve its monitoring of administrative costs by establishing reporting
requirements for administrative costs in its intergovernmental agreements.
The finding of the Auditor General is agreed to and the audit recommendation will be
implemented.
3. The Department of Health Services should update its Accounting and Auditing Procedures
Manual for ADHS Funded Programs, including adding the administrative cost definition and
guidelines.
The finding of the Auditor General is agreed to and the audit recommendation will be
implemented.
4. Once administrative costs are clearly defined and administrative cost reporting requirements are in
place, the Program should improve its accountability for its use of tobacco tax revenues by
developing a plan for measuring the amount of local project funding used for providing program
services.
The finding of the Auditor General is agreed to and the audit recommendation will be
implemented.
Other Performance Audit Reports Issued Within
the Last 12 Months
98-13 Private Enterprise Review Board
98-14 Adult Services
98-15 Podiatry Board
98-16 Board of Medical Examiners
98-17 Department of Health Services—
Division of Assurance and Licensure
98-18 Governor’s Council on Develop-mental
Disabilities
98-19 Personnel Board
98-20 Department of Liquor
98-21 Department of Insurance
98-22 State Compensation Fund
99-1 Department of Administration,
Human Resources Division
99-2 Arizona Air Pollution Control
Commission
99-3 Home Health Care Regulation
99-4 Adult Probation
99-5 Department of Gaming
99-6 Department of Health Services—
Emergency Medical Services
99-7 Arizona Drug and Gang Policy
Council
99-8 Department of Water Resources
99-9 Department of Health Services—
Arizona State Hospital
99-10 Residential Utility Consumer
Office/Residential Utility
Consumer Board
99-11 Department of Economic Security—
Child Support Enforcement
99-12 Department of Health Services—
Division of Behavioral Health
Services
99-13 Board of Psychologist Examiners
99-14 Arizona Council for the Hearing
Impaired
99-15 Arizona Board of Dental Examiners
99-16 Department of Building and
Fire Safety
Future Performance Audit Reports
Department of Health Services—Bureau of Epidemiology
and Disease Control Services
Department of Health Services—Division of Behavioral Health Services’
Sunset Factors
Arizona State Board of Accountancy