Performance audit, Department of Health Services' Tobacco Education and Prevention Program |
Previous | 1 of 2 | Next |
|
|
Small
Medium
Large
Extra Large
Full-size
Full-size archival image
|
This page
All
|
State of Arizona Office of the Auditor General PERFORMANCE AUDIT Report to the Arizona Legislature By Debra K. Davenport Acting Auditor General September 1999 Report No. 99-17 DEPARTMENT OF HEALTH SERVICES' TOBACCO EDUCATION AND PREVENTION PROGRAM The Auditor General is appointed by the Joint Legislative Audit Committee, a bipartisan committee composed of five senators and five representatives. His mission is to provide independent and impar-tial information and specific recommendations to improve the operations of state and local government entities. To this end, he provides financial audits and accounting services to the state and political subdivisions and performance audits of state agencies and the programs they administer. The Joint Legislative Audit Committee Senator Tom Smith, Chairman Representative Roberta Voss, Vice-Chairman Senator Keith Bee Representative Robert Burns Senator Herb Guenther Representative Ken Cheuvront Senator Darden Hamilton Representative Andy Nichols Senator Pete Rios Representative Barry Wong Senator Brenda Burns Representative Jeff Groscost (ex-officio) (ex-officio) Audit Staff Shan Hays—Manager and Contact Person (602) 553-0333 Miriam Seymore—Audit Senior Angelica Gonzalez—Staff Raena Honan—Staff William Parker—Staff Copies of the Auditor General’s reports are free. You may request them by contacting us at: Office of the Auditor General 2910 N. 44th Street, Suite 410 Phoenix, AZ 85018 (602) 553-0333 Additionally, many of our reports can be found in electronic format at: www.auditorgen.state.az.us 2910 NORTH 44th STREET • SUITE 410 • PHOENIX, ARIZONA 85018 • (602) 553-0333 • FAX (602) 553-0051 DEBRA K. DAVENPORT, CPA ACTING AUDITOR GENERAL STATE OF ARIZONA OFFICE OF THE AUDITOR GENERAL September 13, 1999 Members of the Legislature The Honorable Jane Dee Hull, Governor Dr. James Allen, Director Department of Health Services Transmitted herewith is a report of the Auditor General, A Performance Audit of the Department of Health Services’ Tobacco Education and Prevention Program. This report is in response to a May 27, 1997, resolution of the Joint Legislative Audit Committee. This performance audit was conducted as part of the Sunset review set forth in A.R.S. §§41-2951 through 41-2957. This is the fifth in a series of reports to be issued on the Department of Health Services. This report addresses the Program’s need for improved evaluations to measure its effectiveness in preventing youth from starting to use tobacco, encouraging and assisting tobacco users to quit, and reducing exposure to secondhand smoke. We recommend that the Program develop an evaluation strategy and comprehensive plan that corresponds to Program goals, improve evaluation guidance given to the local projects it oversees, and coordinate with other entities to obtain useful tobacco-related information. In addition, the Program needs to improve its guidance of statewide and local projects in their tobacco control efforts. For example, it should develop written policies and procedures, facilitate communication among its statewide and local projects, and fully utilize its Tobacco Use and Prevention Advisory Committee. Further, the Program should monitor local projects’ administrative costs, which will require developing clear guidelines and establishing appropriate reporting requirements. Improved communication and coordination, along with the effective use of resources, will become increasingly important, as the Program’s future funding level represents a decrease of nearly 46 percent from the current level. September 13, 1999 Page –2- As outlined in its response, the Department agrees with all of our findings, and will imple-ment the recommendations. However, the decision to continue the Tobacco Use and Pre-vention Advisory Committee rests with the Legislature. My staff and I will be pleased to discuss or clarify items in the report. This report will be released to the public on September 14, 1999. Sincerely, Debbie Davenport Acting Auditor General Enclosure i OFFICE OF THE AUDITOR GENERAL SUMMARY The Office of the Auditor General has conducted a performance audit of the Department of Health Services’ Tobacco Education and Prevention Program (Program). This audit was conducted pursuant to a May 27, 1997, resolution of the Joint Legislative Audit Committee, and as part of the Sunset review set forth in Arizona Revised Statutes §§41-2951 through 41-2957. This is the fifth in a series of six audits of the Department of Health Services. The Program promotes a tobacco-free Arizona through work in three goal areas: preventing youth from starting to use tobacco, encouraging and assisting tobacco users to quit, and reducing exposure to secondhand smoke. With a staff of 16, the Program oversees tobacco control efforts on both local and statewide lev-els. On the local level, the Program has intergovernmental agreements with each county health department to provide to-bacco cessation counseling, conduct youth awareness programs, and establish community advisory committees. The Program also has intergovernmental agreements with various entities to support statewide tobacco control efforts and complement local activities. Statewide efforts include a tobacco control advertising campaign, the Arizona Smokers’ Helpline, materials and infor-mation clearinghouses, training for tobacco cessation counselors, and evaluations of Program performance. Improved Evaluations Are Needed to Measure Program Effectiveness (See pages 7 through 16) Evaluations at the state and local levels have not yet produced an adequate assessment of the Program’s tobacco control efforts. Arizona Laws 1995, Chapter 275 mandates evaluations to deter-mine whether the Program’s efforts are cost-effective and having any impact on tobacco use. Several of the Program’s attempted evaluations have encountered logistical and design problems. For example, the Program spent $900,000 on an evaluation that had to be terminated because it did not meet the Program’s The Program oversees to-bacco control efforts on both local and statewide levels. Summary ii OFFICE OF THE AUDITOR GENERAL information needs. In addition, the Program has been unable to establish a baseline on tobacco use among youth, which is an essential part of knowing the impact of its advertising campaign and other activities. The Program has had more success in meas-uring the prevalence of adult tobacco use, and expects to issue its first follow-up report on this topic by the end of 1999. More re-cent evaluations, such as a project that will provide a statewide assessment of cessation services by combining evaluation infor-mation from local projects, are more promising but have not yet been completed. Overall, the Program’s evaluation approach to date leaves it far short of knowing whether its programs are working. To improve its ability to evaluate the effect of its statewide and local activities, the Program needs to develop an evaluation strategy, improve its guidance to local project staff, and explore opportunities to coordinate its efforts with other entities that collect relevant data or conduct other studies of the Program’s target populations. The Program only recently hired a statewide evaluator, and as a result now has a better opportunity to make these improvements. The evaluation strategy should incorporate specific goals, objectives, and performance indicators, and it should help identify ways to better collect evaluation data. For example, since at least three other entities are studying youths’ attitudes and behaviors, some schools are declining to participate in school-based surveys. As an alternative, the Program can seek ways of using data collected by others. The strategy should also link statewide and local project efforts to the Program’s outcome objectives, in accordance with Centers for Disease Control guidelines (the federal agency responsible for coordinating to-bacco control efforts). Finally, in assembling its evaluation strat-egy, the Program should seek input from local project staff and incorporate recommendations from a 1998 assessment by the University of Arizona’s Cancer Center. Program Needs to Improve Its Guidance of Tobacco Control Efforts (See pages 17 through 24) The Program needs to improve the guidance and communica-tion it provides to local projects and statewide contractors. The Program has not developed manuals or written policies and The Program has yet to establish a baseline on youth tobacco use. Summary iii OFFICE OF THE AUDITOR GENERAL procedures, but instead waits for local project officials to seek guidance for many day-to-day decisions about their operations. The guidance provided is often inconsistent or untimely. For example, local project staff generally have to wait about three weeks to receive approval to put information about local area events on the statewide Internet Web site. By then, the events have already taken place. Similarly, feedback is often delayed for months on monthly reports submitted by local projects. The lack of clear and timely guidance and communication has resulted in duplicated efforts at events, redundant development of materials, and poor coordination between programs. For ex-ample, lack of communication between the statewide advertising program and the Smokers’ Helpline meant that the Helpline was unprepared to handle a heavy volume of calls when advertise-ments were broadcast that gave the Helpline’s 800 number. Likewise, the Program does not consistently inform local projects about its advertising contractor’s activities. Consequently, both the advertising contractor and local projects have sent crews to the same public events. To effectively guide Arizona’s tobacco control efforts, the Pro-gram needs to change the level and amount of guidance, making it more strategic and system-wide. The first step is to create a strategic plan that would help focus the Program’s efforts and better coordinate its statewide and local project resources. In the future, a strategic plan will become increasingly important, be-cause the Program’s current level of funding will decrease by nearly half as a buildup of reserves from the Program’s earliest years is rapidly spent. In developing a strategic plan and refo-cusing the guidance provided to its contractors, the Program should build on the efforts it has recently begun in seeking input fromits contractors and its advisory committee. Lack of coordination meant that the Helpline was unprepared to handle a heavy volume of calls. Summary iv OFFICE OF THE AUDITOR GENERAL Program Does Not Monitor Administrative Costs (See pages 25 through 28) Although the Program has established a limit of 10 percent on administrative costs, it cannot determine how much of the mon-ies provided for operating local education and prevention proj-ects are used for administration. It has not provided adequate guidelines for identifying such costs, and it has not required local projects to submit expenditure reports that contain enough detail to show which expenditures are administrative. As a result, the Program cannot determine the amount or percentage of Pro-gram monies that are spent on services, such as helping smokers quit and preventing youth from starting to use tobacco. Since services are often provided by several layers of entities charging administrative costs, Program services could account for much less that 90 percent of local project funding after counties, con-tractors, and subcontractors each spend up to 10 percent of their budgets on administrative costs. To ensure that local projects stay within the 10 percent limit, the Program needs to establish clear definitions and more complete reporting requirements. Several layers of entities charge administrative costs. v OFFICE OF THE AUDITOR GENERAL TABLE OF CONTENTS Page Introduction and Background ......................... 1 Finding I: Improved Evaluations Are Needed to Measure Program Effectiveness................................................ 7 Evaluation Requirements............................................... 7 Program Effectiveness Is Not Yet Known........................................................... 8 The Program Can Improve Statewide and Local Project Evaluation Efforts ........................................................... 11 Recommendations .......................................................... 16 Finding II: Program Needs To Improve Its Guidance of Tobacco Control Efforts .............................. 17 Program Structure .......................................................... 17 Program Does Not Provide Adequate Guidance or Communication to Statewide and Local Projects.......................................... 18 Several Factors Contribute to Ineffective Guidance....................................................... 21 Formal Planning and Written Policies Needed to Enhance Program Efficiency...................................... 23 Recommendations .......................................................... 24 Table of Contents vi OFFICE OF THE AUDITOR GENERAL TABLE OF CONTENTS (Concl’d) Page Finding III: Program Does Not Monitor Administrative Costs ................................... 25 The Program Has Not Adequately Defined or Monitored Administrative Costs ................................... 25 Program Needs to Clarify Definitions and Establish Reporting Requirements.................................................................. 27 Recommendations .......................................................... 28 Agency Response Table Table 1 Arizona Department of Health Services Tobacco Education and Prevention Program Statement of Revenues, Expenditures, and Other Changes in Fund Balance Years Ended June 30, 1997, 1998, and 1999 (Unaudited)................................................... 4 1 OFFICE OF THE AUDITOR GENERAL Item 1: Program Goals n Preventing youth from starting to use tobacco; n Encouraging and assisting tobacco users to quit; and n Reducing exposure to secondhand smoke INTRODUCTION AND BACKGROUND The Office of the Auditor General has conducted a performance audit of the Department of Health Services’ Tobacco Education and Prevention Program. This audit was conducted pursuant to a May 27, 1997, resolution of the Joint Legislative Audit Com-mittee, and as part of the Sunset review set forth in Arizona Re-vised Statutes §§41-2951 through 41-2957. This is the fifth in a series of six audits of the Department of Health Services. Tobacco Education and Prevention Program The Tobacco Education and Prevention Program (Program) originated in a 1994 voter initiative called the Tobacco Tax and Health Care Act. The Act increased the tobacco sales tax to 58 cents per pack and required that part of the revenue generated from the increase be used to provide funding for tobacco educa-tion and prevention. In 1995, to implement the Act, the Legisla-ture enacted House Bill 2275, which created the Health Educa-tion Account for tobacco prevention and reduction program and monies and directed the Department of Health Services to ad-minister the Account. The bill also established the Tobacco Use Prevention Advisory Committee, which is required to review Program activities and make recommendations to the Depart-ment. The legislation contains a provision that will terminate the Committee in December 1999. The Program seeks to promote a tobacco-free Arizona through a combination of statewide and local efforts. These efforts target three primary goals (see Item 1). To achieve these goals, the Program has adopted a comprehensive structure recommended by the federal Centers for Disease Control, which includes the following elements: Introduction and Background 2 OFFICE OF THE AUDITOR GENERAL n Local Projects—Intergovernmental agreements with each county health department support local efforts, such as to-bacco cessation counseling, youth awareness programs, and community partnerships established with local not-for-profit organizations. n Statewide Advertising Campaign—Under a competitive contract with an advertising company, the Program supports a highly visible, statewide tobacco control advertising cam-paign. The nationally recognized campaign employs a vari-ety of creative approaches including television and radio commercials, print advertisements, a merchandise center for campaign-related materials, and a mobile “Ash Kicker” vehi-cle with tobacco control displays aimed at youth. n Statewide Support Services—The Program provides funding and support for the Arizona Smokers’ Helpline, three information and materials resource clearinghouses, a Web site, and other services. These services are provided through intergovernmental agreements with Arizona’s three state universities. n Statewide Cessation and Evaluation Services—The Ari-zona Cessation Training and Evaluation Program (ACTEV), a Program-funded partnership between the University of Arizona and local projects, provides tobacco cessation certifi-cation and evaluation training for local projects and healthcare providers. The certification training focuses on skills required to encourage and assist tobacco users to quit and remain tobacco-free. The evaluation training teaches the skills necessary to evaluate local projects’ tobacco cessation activities. n Local and Statewide Enforcement Programs—The Pro-gram funds efforts to ensure compliance with laws restricting tobacco sales to minors. State and federal monies are allo-cated for enforcement. The Program receives grant monies through the federal Food and Drug Administration (FDA) enforcement program. The federal monies are administered by the Program and provided to communities for enforce-ment. Additionally, state-funded local projects in some areas, such as the City of Mesa, currently include enforcement in their activities. Introduction and Background 3 OFFICE OF THE AUDITOR GENERAL Budget, Personnel, and Organization The Program is primarily funded through state tobacco taxes with some additional funding from federal grants. The Program receives 23 percent of each dollar derived from the 58-cent-per-pack Arizona tobacco sales tax. The Program’s share of fiscal year 1998-99 tobacco tax revenue amounted to approximately $26.5 million. In addition, the Program has approximately $46 million in available carryover funds and interest earned, since the Legislature capped Program spending at $10 million and $15 million during its first and second years of operation, respec-tively. The Legislature later authorized an additional $9 million in available monies for the Program’s second year. (See Table 1, page 4.) Currently, the Legislature restricts Program spending to no more than 95 percent of the previous year’s tobacco tax reve-nues. For fiscal year 1998-99, the Program anticipates expenditures of approximately $32.7 million from state monies, including $1.25 million in transfers to the Arizona Department of Education for tobacco education in the public schools. The Program funds school-based tobacco control education efforts in accordance with federal Centers for Disease Control recommendations. Pro-gram expenditures primarily consist of services provided at the statewide and local projects level. A state Program office admin-isters the tobacco education and prevention projects as follows: The Program office, housed within the Division of Public Health Services, is authorized 16 staff. The staff are directed by an office chief and structured into major program areas, including local projects support, statewide support, financial management, marketing, and evaluations. Office administration expenditures for fiscal year 1998-99 were $726,000. Statewide functions are performed mainly through a contract with an advertising company and intergovernmental agree-ments with all three state universities. Expenditures for such services were approximately $21.4 million in fiscal year 1998-99, including $13.5 million for the advertising campaign contract. The Program anticipates expenditures of approxi-mately $32.7 million from state monies. Introduction and Background 4 OFFICE OF THE AUDITOR GENERAL Table 1 Arizona Department of Health Services Tobacco Education and Prevention Program Statement of Revenues, Expenditures, and Other Changes in Fund Balance Years Ended June 30, 1997, 1998, and 1999 (Unaudited) 1997 1998 1999 Revenues: Tobacco taxes $27,251,281 $26,779,642 $26,457,461 Interest on investments 2,145,792 2,745,938 2,777,685 Federal grants and contracts 48,501 81,962 264,964 Other 2,653 Total revenues 29,445,574 29,607,542 29,502,763 Expenditures: Personal services 402,929 393,543 461,315 Employee related 81,589 84,280 95,227 Professional and outside services 1 13,181,941 16,250,403 17,032,162 Travel, in-state 18,936 9,703 11,994 Travel, out-of-state 11,262 9,005 11,629 Aid to organizations 2 4,394,847 7,405,239 14,960,702 Other operating 107,725 50,200 112,442 Capital outlay 25,269 37,026 2,354 Allocated costs 10,360 21,074 30,898 Total expenditures 18,234,858 24,260,473 32,718,723 Excess of revenues over(under) expenditures 11,210,716 5,347,069 (3,215,960) Fund balance, beginning of year 32,437,568 43,648,284 48,995,353 Fund balance, end of year 3 $43,648,284 $48,995,353 $45,779,393 1 Includes $11.5, $13.2, and $13.5 million expended for media and sports contracts in 1997, 1998, and 1999, respectively. 2 Includes $3.1, $6.1, and $10.6 million passed through to counties for local projects in 1997, 1998, and 1999, respectively. In addition, includes $2.9 million passed through to Arizona universities in 1999 for statewide functions and $1.25 million passed through to the Arizona Department of Education for to-bacco education at the public schools in 1998 and 1999. 3 The Department has legislative guidelines for spending its fund balance. Source: The Arizona Financial Information System Revenues and Expenditures by Fund, Program, Organi-zation, and Object and Trial Balance by Fund reports; and Department-provided financial infor-mation for the years ended June 30, 1997, 1998, and 1999. Introduction and Background 5 OFFICE OF THE AUDITOR GENERAL Local projects funded by the Program through intergovernmen-tal agreements are operated by counties, tribal governments, and their subcontractors. These enti-ties were allocated about $13.7 million in fiscal year 1998-99 to develop activities aimed at get-ting communities, coalitions of concerned citizens, and interested groups involved in tobacco control goals and efforts. Maricopa County administers its local projects’ activities through subcontracts for programs and services while the remaining 14 counties directly administer the projects. Audit Scope and Methodology A combination of methods was used to study the issues ad-dressed in this audit. For example, Auditor General staff: n Reviewed the minutes from a sample of 8 Tobacco Use Pre-vention Advisory Committee meetings held from 1995 to 1998 and interviewed 7 of the 10 Committee members. n Reviewed contracts, intergovernmental agreements, and evaluations for statewide and local projects. Interviews were conducted with 15 members of local projects staff represent-ing all 15 Arizona counties, in addition to 7 local projects’ subcontractors. n Attended meetings of the Coalition for a Tobacco Free Ari-zona, the Program’s 4th annual conference, and regional meetings with the local projects held by the Program. n Interviewed representatives from all the organizations pro-viding statewide support services including the advertising campaign contractor, the Arizona Tobacco Information Net-work, the Arizona Prevention Resource Center, and the Ari-zona Cessation Training and Evaluation Program. Item 2: Tobacco Settlement In the future, Arizona expects to receive substantial funding from the multi-state tobacco industry settlement. However, the use of such funds has not been determined by the Gov-ernor and the Legislature. The future impact of settlement dollars on the Program fund-ing is unknown. Introduction and Background 6 OFFICE OF THE AUDITOR GENERAL n Conducted an extensive review of literature, including jour-nal articles, books, Internet Web sites, and reports from other states. n Interviewed administrators of similar programs in 7 other states.1 This audit contains findings and recommendations in three ar-eas: n The need to improve the Program’s ability to evaluate the success of statewide and local projects’ tobacco control ef-forts; n The need to improve the Program’s oversight of local proj-ects’ efforts; and n The Program’s inability to monitor local project administra-tive costs. This audit was conducted in accordance with government auditing standards. The Auditor General and staff express appreciation to the Di-rector of the Department of Health Services, the Assistant Direc-tor for the Division of Public Health Services, the management and staff of the Tobacco Education and Prevention Program, the management and staff of contracted statewide support services, and local project staff for their cooperation and assistance throughout the audit. 1 Other states contacted were California, Florida, Massachusetts, Min-nesota, Mississippi, Oregon, and Texas. These states were selected be-cause they had funding sources and goals similar to Arizona’s. 7 OFFICE OF THE AUDITOR GENERAL FINDING I IMPROVED EVALUATIONS ARE NEEDED TO MEASURE PROGRAM EFFECTIVENESS The Program has not adequately assessed the effectiveness of its tobacco prevention and cessation efforts. Improvements in statewide and local evaluations would result in better informa-tion about the Program’s success. The Program’s newness, to-gether with delays in hiring an evaluator to coordinate the vari-ous evaluation efforts, contributed to the slow start. An evaluator is now in place, giving the Program the opportunity to address past problems. To maximize the value of evaluation efforts, the Program should work with national experts and local projects to develop an evaluation strategy, including a plan to provide out-come measures and other data useful to state and local project staff. Evaluation Requirements Laws 1995, Chapter 275 requires biannual program evaluations to determine whether the Program’s education, prevention, and advertising campaigns are cost-effective and having an impact on tobacco use. Program evaluations should include the results of all monitoring of tobacco-related behaviors, attitudes, and health outcomes assessed at regular intervals. In following the Centers for Disease Control’s tobacco control program model, Arizona’s Program requires evaluations on both the statewide and local project levels. As part of their intergovernmental agreements with the Program, local projects are required to assist with data collection, including monthly and quarterly reports that may be monitored and accumulated for ongoing program evaluation. Local projects are also required to specify the evalua-tion budget and staffing they will use to conduct Program evaluations. Finding I 8 OFFICE OF THE AUDITOR GENERAL Program Effectiveness Is Not Yet Known While the Program has made some efforts to measure the suc-cess of statewide efforts over the last three years, its effectiveness has not been determined. Evaluation efforts since the Program’s 1995 inception have yielded inconclusive results; however, the Program is currently working on several evaluations that may provide more useful information in the future. Evaluation efforts to date have not determined effectiveness— The Program has attempted several evaluations, including a proc-ess evaluation that was expected to track both its short-term and long-term progress; an evaluation aimed at assessing the effec-tiveness of the statewide advertising campaign; and a survey intended to measure changes in tobacco use. However, logistical and design problems hampered the success of two of these ef-forts, and the third has not yet produced meaningful results. n Process Evaluation—The Program attempted a process evaluation of local project activities, but discontinued the ef-fort when the evaluation failed to provide the information needed. In 1996, the Program contracted with an organiza-tion for an evaluation it called Open Systems. The Open Systems effort was designed to examine the quantity and quality of activities performed by the local projects, and their progress toward Program goals. However, the evalua-tion’s design did not incorporate centralized planning. As a result, the design did not allow it to compare local projects to one another or to compare their effectiveness over time, and did not use random sampling or other methods in-tended to ensure valid results. Further, local project person-nel reported many problems with the evaluation, including the amount of time required by the process and frequent changes in the way they were instructed to implement the evaluation. Overall, the Program expended approximately $900,000 on a process evaluation that did not achieve its goals. n Youth Tobacco Survey—The Program did not obtain in-formation on youth tobacco use before starting its advertising campaign, and it struggled in subsequent efforts to measure youth’s use of and exposure to tobacco. The Arizona De- The Program expended approximately $900,000 before discontinuing a process evaluation that did not achieve its goals. Finding I 9 OFFICE OF THE AUDITOR GENERAL partment of Health Services’ Bureau of Public Statistics con-ducted a statewide telephone survey on youth smoking during 1996 and 1997, after the advertising campaign aimed at youth had begun. The survey attempted to estimate the prevalence of tobacco use, identify knowledge about access to tobacco products, and identify exposure to secondhand smoke. However, the survey’s timing undermines its poten-tial usefulness in measuring the Program’s impact, since the advertising campaign had already begun. In addition, the survey may have underestimated the prevalence rate of youth smoking, because it required parental consent and the youths may not have given honest answers in the presence of their parents. In order to determine the Program’s impact, a survey must be repeated to measure whether prevalence has changed from the first time the survey was administered. However, the Program is replacing the existing survey with another survey using a different methodology. While this will further delay the Program’s ability to measure its effectiveness, the new survey should produce more accurate information on youth tobacco use. n Advertising Evaluation—An evaluation of the statewide advertising campaign may yield more useful results, but key components will not be completed until September 1999. The first component of this advertising evaluation included a historical report to the Legislature, completed in October 1998, and a study of the cost-effectiveness of the advertising campaign, completed in January 1999. The October 1998 report incorporated a random sample telephone survey to measure exposure to and recognition of the 1996, 1997, and early 1998 advertisements and re-ported changes in attitudes and behaviors during the first years of the campaign. However, the January 1999 report could not fully answer questions about the cam-paign’s cost-effectiveness. According to Program offi-cials, Arizona was the first state that attempted to meas-ure cost-effectiveness of its advertising campaign. Using A.C. Neilsen (television) and Arbitron (radio) data, the study measured the number of people who saw or heard television, radio, billboard, and other advertisements. The study also incorporated results of adult and youth An evaluation of the statewide advertising campaign may yield more useful results. The Program still lacks a baseline on the prevalence of youth smoking. Finding I 10 OFFICE OF THE AUDITOR GENERAL smoking prevalence surveys to determine whether peo-ple remembered seeing the advertisements. It found, for example, that the cost per person in the adult population who saw television advertisements was $.01, and the cost per adolescent who remembered seeing the television advertisements was $25.66. However, the study was not designed to measure whether people had changed their smoking behaviors or attitudes. A yet-to-be-completed outcome component may yield more useful results. In order to look more closely at the link be-tween the advertising campaign and changes in attitudes and behaviors, the Program contracted with the Arizona Cancer Center at the University of Arizona, in 1998, to con-tinue evaluating the advertising campaign. Once completed, the evaluation’s outcome component will include six waves of surveys, including an in-depth survey of youth in schools, in-depth telephone interviews, and a random tele-phone sample of Hispanic households with women of childbearing age. The first wave of the outcome component was completed in late 1998. The fifth and sixth waves of surveys are due September 30, 1999, and were not com-pleted by the end of audit fieldwork. To date, the Program has expended approximately $265,000 for the advertising evaluation. Future evaluations may provide better information—In addition to the previously mentioned evaluations, the Program is cur-rently working on other evaluations that may provide more useful information in the future. However, it is too early to de-termine if these evaluations will yield desired outcomes. n Adult Tobacco Survey—The Program has had more suc-cess in its efforts to measure the prevalence of adult tobacco use than it has had with youth, although it has not yet measured any changes in prevalence of use since the Pro-gram’s inception. The initial survey, conducted in 1996 by the Department’s Bureau of Public Statistics, yielded infor-mation on smoking prevalence, individual tobacco use his-tories, and exposure to secondhand smoke. It also gathered information on reported tobacco control counseling pro-vided by health care professionals, and measured aware-ness of the Program’s tobacco control advertising campaign. The Program has had more success in its efforts to measure adult tobacco use. The cost-effectiveness study provided very limited results. Finding I 11 OFFICE OF THE AUDITOR GENERAL The Bureau of Public Statistics plans to repeat the survey and expects to release a report on the results in September 1999. This will be the first time that the Program can identify any changes in smoking prevalence since it began. n Project evaluations—Evaluation of several Program-funded projects should show whether or not those projects achieve their goals. For example, the Arizona Cessation Training and Evaluation Project (ACTEV) is a Program-funded partnership between the University of Arizona and local projects to help them set up and evaluate cessation services for tobacco-dependent community members. ACTEV will use a standardized questionnaire and provide evaluation feedback to local projects on the effectiveness of their cessation activities. ACTEV will also conduct a longitudinal study to track 500 people over time, a study to examine why people drop out of cessation activities, and a third study to at-tempt to gauge the power of brief interventions, using a random sample of the general population. Similarly, some other projects will include evaluations. CHAMPS, a tri-university partnership that trains youth to be “CHAMPS” by resisting tobacco, includes an evaluation compo-nent. In this project, participating youth will take tests at the beginning and end of the program to determine what knowl-edge they gained and whether they changed their attitudes or behavior. Finally, a pilot project for providing prevention coun-seling to Women, Infants, and Children (WIC) participants in four counties also includes an evaluation component to assess the intervention and its effect on the target population. The Program Can Improve Statewide and Local Project Evaluation Efforts While several factors have contributed to the Program’s inability to measure its effectiveness to date, the Program currently has the opportunity to address those factors in order to better assess its progress in the future. The Program recently addressed a Finding I 12 OFFICE OF THE AUDITOR GENERAL major deficiency in its evaluation structure by hiring a statewide evaluator. With the new evaluator in place, the Program should be able to develop an overall evaluation strategy and provide better direction and technical assistance to local projects. The Program should also explore opportunities to gain efficiencies by working with other entities to obtain needed information. Lack of evaluator, Program’s newness contributed to past evaluation problems—Delays in hiring a statewide evaluator contributed to the lack of continuity and focus for the Program’s evaluations. Until April 1999, the Program operated without a statewide evaluator. With the evaluator position now filled, Pro-gram officials anticipate improvements in both statewide and local project evaluations. Because officials lacked evaluation expertise and because na-tional state-operated tobacco control efforts were new, the Pro-gram made early mistakes that contributed to the difficulties encountered in evaluations to date. For example, eagerness to implement key parts of the Program, such as the advertising campaign, led directly to the Program’s inability to establish baseline information describing the prevalence of tobacco use among the campaign’s target populations before the advertise-ments began to appear. Similarly, inexperience with evaluation, and a lack of national models to learn from, made it difficult for management to predict that the process evaluation would not meet its needs. The Program should focus and refine its evaluation strategy— As a first step in improving its approach to evaluations, the Pro-gram should develop an evaluation plan to guide statewide and local efforts. Currently, the Program’s evaluation efforts lack continuity and focus, because no evaluation plan exists to help guide statewide or local project staff. A strategic plan is necessary to provide specific goals, objectives, and performance indicators, and it can also identify ways to better collect data. The new statewide evaluator intends to develop an evaluation plan by the end of 1999. In doing so, the evaluator should also consider ob-taining input from statewide, local project personnel, and other stakeholders in determining the future of Program evaluations. In developing this plan, the Program should consider the rec-ommendations made in a 1998 needs assessment conducted by The Program should develop an evaluation plan to guide statewide and local efforts. Finding I 13 OFFICE OF THE AUDITOR GENERAL the University of Arizona Cancer Center under a contract with the Program. This $25,000 study, specifically intended to identify ways to improve evaluations, created a profile of local projects’ current evaluation efforts and made numerous specific recom-mendations. However, the Program has not implemented the study’s recommendations for improving its overall evaluation strategy, such as forming a regional evaluation team, changing process evaluations, and improving local project efforts. Other study recommendations pertained to the kinds of evaluations the Program should conduct in the future, such as another proc-ess evaluation. The Program should improve its evaluation guidance to local projects—At the same time it improves its statewide evaluation plan, the Program needs to improve its oversight of local project evaluation efforts. Although local projects must include an evaluation component in their programs, the Program has not effectively communicated its evaluation expectations to local projects. Also, to maximize the usefulness of future local project evaluation efforts, the Program should link local project per-formance indicators to statewide goals. The Program needs to better communicate its evaluation expec-tations to local projects, as well as improve coordination between local project and statewide evaluation efforts. According to local project coordinators and evaluators, communication between the Program and local projects is inadequate. Local projects repeat-edly stated that the Program has not provided adequate evalua-tion guidance or technical assistance. Further, the Program does not have standardized evaluation practices that could allow local projects to work together or compare themselves to each other. Local projects want the new Program evaluator to provide more evaluation guidance and technical assistance by holding state-wide evaluation meetings and possibly creating an evaluation manual that would outline evaluation activities. Local projects also want the evaluator to provide feedback on whether their efforts are meeting statewide expectations and contributing to a reduction in tobacco use. According to the Centers for Disease Control, the federal agency that provides guidance to states in implementing a comprehen-sive approach to preventing and controlling tobacco use, pro-gram evaluation efforts should build upon and complement Communication between the Program and local projects is inadequate. Finding I 14 OFFICE OF THE AUDITOR GENERAL tobacco-related evaluation systems by linking statewide and local project efforts to outcome objectives. Currently, the Pro-gram has no structure in place that links performance indicators to statewide objectives. The Program could benefit from external opportunities and in-formation— Finally, after focusing its evaluation strategy and providing evaluation guidance, the Program can improve its efforts by working more collaboratively with related entities and developing and collecting better data. While the Program has attempted to coordinate evaluation efforts in the past, many opportunities to improve efficiencies still exist. Specifically, the Program can improve its evaluation efforts by working with other research entities, coordinating the administration of school-based surveys, and gathering available information about to-bacco. n Coordination with other entities—The Program could col-laborate with other entities, such as the Disease Control Re-search Commission, to obtain additional research related to tobacco. The Commission is a state agency whose purpose is to protect public health and safety by contracting with indi-viduals and organizations to conduct research into the cause and prevention of diseases, including behavioral changes such as smoking cessation. Currently, the Program’s Office Chief represents the Director of the Department of Health Services on the Commission. Because Commission funding derives from tobacco sales tax revenues, all Commission contractors are researching tobacco-related diseases in Ari-zona. Both the Program and the Commission could poten-tially benefit from collaborating on evaluations. For example, Massachusetts, another state with a program like Arizona’s, benefits from such research by contracting out small research projects to universities and other community researchers. Approximately five research projects per year are contracted out to fill in gaps created by other surveys or to follow up on problems or issues identified in certain populations. Recent projects include analyses of special population groups, in-cluding immigrants and elderly minorities. n Coordination on school-based surveys—According to local project coordinators and other evaluators, an excessive number of surveys are conducted in Arizona’s schools. As a Finding I 15 OFFICE OF THE AUDITOR GENERAL result, some school districts have declined to participate in school-based surveys. To reduce this saturation of surveys, the Program should continue to work with other entities that already obtain information about youth through school-based surveys, such as the Arizona Criminal Justice Commis-sion. The Program is currently coordinating with Behavioral Health Services on the High Risk Children’s Survey, which will include tobacco questions. n Other available information—The Program should work with other entities to gain a better understanding of tobacco use in Arizona. Other states’ tobacco programs compile in-formation gathered by various entities. For example, Oregon, California, and Massachusetts use their Departments of Revenue to collect tobacco consumption data from tobacco sales taxes. In addition, Oregon obtains birth and death in-formation from their vital statistics office to determine whether tobacco was used during a pregnancy or whether tobacco contributed to the death of a person. Oregon also uses data collected from the Oregon Office of Alcohol and Drug Abuse Program. Finding I 16 OFFICE OF THE AUDITOR GENERAL Recommendations 1. The Program should develop an evaluation strategy, which includes a comprehensive evaluation plan that corresponds to Program goals. As part of its evaluation strategy, the Pro-gram should: a. Improve planning by seeking the input of national experts and other state officials regarding ways to establish linkages be-tween statewide and local efforts; b. Review and implement recommendations from the University of Arizona Cancer Center’s Needs Assessment Study to help refine Program activities; c. Establish a youth smoking prevalence baseline; and d. Continue efforts to work with the CDC on implementing the new Youth Tobacco Survey. 2. The Program should improve its communication with local projects by: a. Seeking recommendations for improvement from local project coordinators and evaluation staff; b. Providing feedback to local project personnel regarding evaluation reports and requirements; and c. Sharing evaluation information gathered by local projects to help improve local-level evaluations. 3. The Program should attempt to obtain tobacco-related in-formation from other entities, such as: a. Tobacco consumption data from the Department of Revenue; b. Birth and death statistics from the Office of Vital Statistics; and c. Tobacco-related studies from the Disease Control Research Commission to help the Program meet its goals. 4. The Program should continue to coordinate surveys with other entities, particularly existing school-based surveys. 17 OFFICE OF THE AUDITOR GENERAL FINDING II PROGRAM NEEDS TO IMPROVE ITS GUIDANCE OF TOBACCO CONTROL EFFORTS The Program needs to improve its interactions with local projects and statewide entities. The Program does not provide adequate guidance or communication to its statewide and local projects, which limits its ability to effectively coordinate its resources. These problems stem from an approach that emphasizes cen-tralized approval and does not take advantage of general guid-ance that could be provided by the Program’s advisory commit-tee. Improvements are needed to increase efficiency, particularly since much less money will be available once the Program has spent the surplus funds that accumulated due to spending re-strictions during its first two years of operation. Program Structure The Tobacco Education and Prevention Program oversees and coordinates Arizona’s tobacco control campaign through local and statewide efforts. To operate tobacco control local projects, the Program has intergovernmental agreements with all Arizona county health departments. These local projects have four duties: n develop a local coalition of individuals and groups and write a plan to support the Program goals; n conduct prevention programs; n provide cessation services for tobacco users; and n promote tobacco-free schools, homes, and workplaces. Local projects vary in their sophistication. Since some local proj-ects began their efforts at different times, they differ in the num-ber and levels of services provided. For example, one of the first local projects, Pima County, began operations in 1996 and has staff and contractors working in all four Program areas. In Finding II 18 OFFICE OF THE AUDITOR GENERAL contrast, the La Paz local project was initiated in 1998 and has just begun to initiate tobacco control activities after spending its first year engaged in planning. In addition, during the first three years of operation, the Director of the Department of Health Services restricted Program efforts to prevention programs for youth and cessation services for pregnant women and their partners. However, some communities used other resources and initiated activities outside these restrictions. In addition to local project efforts, the Program also oversees contractors who conduct statewide tobacco control efforts. Statewide programs include an tobacco control advertising cam-paign, an tobacco control literature and materials clearinghouse, the Smokers’ Helpline, and cessation specialist training. State-wide efforts are intended to complement local project activities. . Program Does Not Provide Adequate Guidance or Communication to Statewide and Local Projects The Program has not adequately guided the tobacco control efforts of its statewide and local projects. While the Program has been in operation for four years, local projects lack basic operat-ing guidelines and timely assistance. Further, the lack of Pro-gram guidance and communication hinders the effective coordi-nation of resources among local projects and statewide entities. The Program does not provide needed guidance or adequate communication to local projects—The Program has not pro-vided consistent and timely technical assistance to statewide and local projects. Because tobacco education and prevention pro-grams are new, local projects rely on Program staff to provide guidance and clarify expectations. However, the Program has not met local projects’ needs with timely information and feed-back. Specific problems include: n No clear policies explaining expectations—The Program has not written policies and procedures or developed a man-ual for local projects to follow. As a result, local projects have received inconsistent instructions regarding Program opera-tions. For example, when the Program directed local projects to develop cessation programs for tobacco users who wished Local projects lack basic operating guidelines and timely assistance. Finding II 19 OFFICE OF THE AUDITOR GENERAL to quit, it had no established guidance for local projects to use in setting certification standards for cessation specialists. As part of the Arizona Cessation Training and Evaluation Proj-ect, a statewide effort to train cessation specialists was just getting underway. Local projects received changing and con-flicting instructions until certification standards were written several months after the training program began. n Untimely answers and approvals—The Program has been slow to respond to local projects’ requests for information and assistance. For example, local project staff cannot submit updated public information about local area events and community services through the statewide Internet Web site without Program office approval. However, approval rou-tinely takes three weeks, by which time the information is out of date. Similarly, it took the Program six months to approve an Internet mailing list and discussion forum that would help local projects communicate with each other. n Lack of feedback—The Program has not provided adequate feedback to its local projects. One valuable resource for feed-back is the monthly reporting from local projects submitted for Program staff review and comment. However, the Pro-gram usually has at least a six-month backlog of reports waiting for review, before feedback can be given to local projects. Most guidance given to local projects is conducted informally through e-mail and verbal updates that Program staff deliver at regional meetings, unsupported by written policies or guidelines. Lack of guidance and communication hinders effective coordina-tion of resources—The lack of Program guidance and communi-cation results in an ineffective use of resources. Problems created include duplication in staffing and materials; incomplete coordi-nation of tobacco control efforts with schools; and a lack of coor-dination among the Program’s advertising contractor, other statewide contractors, and local projects. n Duplicate staffing—The Program does not consistently inform local projects about its advertising contractor’s activi-ties. Consequently, both the advertising contractor and local projects have sent crews to the same public events. The lack of written poli-cies and procedures has led to inconsistent deci-sions. Finding II 20 OFFICE OF THE AUDITOR GENERAL n Duplicate materials—Both state and local projects are dupli-cating each other’s efforts in creating similar tobacco control materials. Tobacco prevention and education programs can-not function without accurate and professional public infor-mation. Although the Program contracts with Arizona State University to operate a statewide clearinghouse to create materials, some local projects are unaware of these efforts. The statewide clearinghouse is currently creating “quit kits” and three designs for decals that announce a smoke-free en-vironment. Since local projects are not regularly involved with the clearinghouse, they are creating their own “quit kits” and smoke-free decals. If the Program involved local projects in such clearinghouse activities, the local projects would not only be aware of clearinghouse efforts, they could participate in the development of materials suited to local needs. n The Program does not facilitate use of its monies in schools—The Program has not facilitated communication with schools regarding the use of tobacco control education monies. Specifically, the Program has not enforced the provi-sions of its interagency agreement with the Arizona Depart-ment of Education, which requires performance standards, monitoring, and regular reports on the use of Program to-bacco control monies provided to schools. Although school settings are crucial for tobacco use prevention, some schools are unaware that some of the monies they receive from ADE are Program monies to pay for tobacco control efforts. In ad-dition, the Program and ADE do not require schools receiv-ing these monies to work with statewide and local projects. As a result, schools have purchased curricula or attempted to develop it themselves, unaware they can obtain it without cost through the Program’s statewide clearinghouse. n The Program does not adequately coordinate statewide and local project activities—Crucial information is not shared between statewide programs, or between these pro-grams and local projects, making it more difficult for state-wide and local project staff to carry out their duties effec-tively. Experiences with the Smoker’s Helpline, a statewide program, provide an illustration of these problems. One set of problems involved the lack of coordination between Helpline and another statewide program, the advertising State and local projects are duplicating each other’s efforts. Crucial information is not shared. Finding II 21 OFFICE OF THE AUDITOR GENERAL campaign. The Helpline is staffed by counselors who fill re-quests for written material about tobacco use and advise smokers who wish to quit. The Helpline has not been notified in advance when its toll-free number is broadcast on radio and television stations throughout Arizona. Such notification would give Helpline managers an opportunity to prepare for the additional calls such announcements can bring. On sev-eral occasions after such announcements, the Helpline re-ceived hundreds of calls without enough counselors to han-dle them. Likewise, the Helpline has not been notified when Spanish-speaking commercials are aired, giving the staff no opportunity to add enough bilingual counselors. A second set of problems involved coordination between Helpline and local programs. The Program has not permitted the Helpline to refer callers to the local projects in their communities be-cause the Program has not decided if, or how, referrals should be made. As a result, callers may not be aware of Pro-gram services provided by their local project. Several Factors Contribute to Ineffective Guidance Two main factors have limited the Program’s ability to guide tobacco control efforts. First, Program management has overem-phasized supervision of contractors’ day-to-day operations at the expense of establishing broad policies and creating an effective network for open communication among its contractors. Second, the Program has not sought input from its advisory committee, which could help in developing such policies. Program has focused on supervising daily operations—Instead of developing clear guidelines for contractors to follow, the Pro-gram has required contractors to seek guidance and approval for all day-to-day operational decisions. Program management con-sidered such control necessary to establishing a new program, especially with inexperienced staff working at the local projects and to ensure that only appropriate activities received funding. The Program’s supervisory approach may also have contributed to its failure to obtain advice from local project staff on ways to improve the Program. Although Program officials occasionally Finding II 22 OFFICE OF THE AUDITOR GENERAL asked local project staff to fill out questionnaires, or to voice their concerns after hearing a presentation on Program decisions made by the officials, these efforts were inadequate for obtaining meaningful local project input. Recently, however, the Program has improved its efforts to communicate and to solicit input. In regional meetings held early in 1999, local project staff were asked to comment on a proposed form revision and to provide their views regarding how the Smokers’ Helpline might refer callers to the local projects. Limited input sought from advisory committee—The Program has not sought input from its advisory committee, which could assist it in developing broad policies to guide local projects and statewide contractors in their activities. The Tobacco Use Pre-vention Advisory Committee (TUPAC) is an advisory group of physicians, community leaders, and policy makers created by 1995 legislation. TUPAC is chaired by the director of the De-partment but has no decision-making authority. The Program does not consult them prior to making decisions, but TUPAC members are given information and may ask questions. During the audit, five of the public members of TUPAC were inter-viewed. Most report they do not interact with local projects and are unfamiliar with them. In addition, the majority of TUPAC’s public members expressed confusion over their role and some are reluctant to continue as members because they do not feel effective. The Legislature has recognized the need for stronger external oversight. In May 1999, the Legislature passed House Bill 2481. As part of this Department of Health Services’ omnibus bill, TUPAC would have been replaced by a Tobacco Revenue Use Spending and Tracking Commission. This legislation would have given the Commission many of the Program’s current oversight functions, such as developing a five-year plan for the use of monies in the Program’s Health Education Account, per-forming evaluations and reviews of programs and projects funded from the Account, and approving expenditures of mon-ies from the Account. However, this legislation was subse-quently vetoed by the Governor. Without new legislation, TUPAC will sunset on December 31, 1999. The Legislature has recognized the need for stronger external over-sight. Finding II 23 OFFICE OF THE AUDITOR GENERAL Formal Planning and Written Policies Needed to Enhance Program Efficiency In order to improve, the Program needs to change the level and amount of guidance it provides to statewide and local projects. The Program has lacked long-term strategic planning, and has not developed written policies and procedures. Advance plan-ning is essential to establish productive working relationships among statewide and local projects and to effectively coordinate resources. The Program needs to create a strategic plan for statewide and local projects. As noted earlier, lack of understanding of roles, responsibilities, and goals has created problems among local projects and statewide entities. A strategic plan could focus ef-forts of the Program and its statewide and local projects and help effectively coordinate these resources. As part of a strategic plan to guide tobacco control efforts and effectively coordinate re-sources, the Program should determine how to cope with de-creased availability of monies in the future. Currently, the Pro-gram is spending a previously unused surplus of approximately $43.6 million from the Health Education Account. By compari-son, Program management projects future tobacco tax revenues will generate approximately $20 million annually. The Program’s future funding level represents a decrease of nearly 46 percent of its current spending. (For more information on Program fund-ing, see Introduction and Background, pages 1 through 6.) The Program needs to develop written policies and procedures or manuals. Written guidelines would lessen the problem of local projects constantly seeking guidance from Program staff and management. Currently, Program guidelines are contained in separate documents and locations, including intergovern-mental agreements, annual renewal applications, and day-to-day e-mail responses to local project questions. The latter have been stored in the Program’s staff computers since 1995. Finding II 24 OFFICE OF THE AUDITOR GENERAL Recommendations 1. The Program should conduct strategic planning with its ad-visory committee, statewide entities, and local projects, to in-clude: a. Program goals and objectives; b. Allocation of resources; and c. Coordination of Program entities. 2. The Program should establish clear policies and procedures for statewide and local projects to define responsibilities, roles, and restrictions. 3. The Program should aggressively facilitate communication among statewide entities, local projects, and schools in order to share information and coordinate resources. 4. The Program should fully use the talents and experiences of its advisory committee and local projects staff to refine its ac-tivities. 5. The Legislature should consider whether to continue the Tobacco Use Prevention Advisory Committee (TUPAC). 25 OFFICE OF THE AUDITOR GENERAL FINDING III PROGRAM DOES NOT MONITOR ADMINISTRATIVE COSTS The Program cannot determine how much of the monies pro-vided for operating local education and prevention projects are used for administration. While its intergovernmental agreements restrict the amount of Program monies that can be spent on ad-ministrative costs, the Program cannot determine whether local projects adhere to the limit. In order to assess compliance with the limit and, eventually, measure the amount of local project funding spent on services, the Program needs to develop clear guidelines for determining administrative costs and establish appropriate reporting requirements. The Program Has Not Adequately Defined or Monitored Administrative Costs The Program has no assurance that administrative costs are within the limit it has established. The Program has not provided adequate guidelines for determining what should be considered as administrative costs and has not required local projects to submit detailed reports. As a result, the Program cannot deter-mine what portion of its monies has been spent on Program services. Program has established limit on administrative costs—The Program has established a limit on how much of its monies that local projects can spend on administrative costs. The Program's intergovernmental agreements restrict local projects to spending no more than 10 percent of budgeted monies on such costs. This is similar to state agencies, which typically spend 8 to 10 percent of their operating budgets on administrative costs. Program has not developed adequate guidelines for determining such costs—Despite setting the limit, the Program has not pro-vided clear guidelines for measuring administrative costs. The Program has not provided adequate guide-lines for determining administrative costs. Finding III 26 OFFICE OF THE AUDITOR GENERAL Its intergovernmental agreements instruct local projects to follow the standards and practices of the Department's 1987 accounting manual. However, this manual does not provide guidelines for administrative costs. Further, the Program uses the term “indi-rect costs” interchangeably with “administrative costs,” although the two terms have slightly different meanings. According to A Dictionary for Accountants,1 administrative costs are expenses incurred “in the general direction of an organization as a whole.” Indirect costs, according to the Department’s manual, include any costs incurred for a joint purpose and benefiting more than one program. This mix of terms further adds to the potential con-fusion about which costs should be included. Program does not require detailed reporting—Even if the Pro-gram provided clear guidelines regarding administrative costs, it could not determine whether local project administrative expen-ditures exceed the 10 percent limit because it does not require detailed tracking and reporting of costs. The Program requires financial reports showing total expenditures in categories such as personal services (salaries) and employee-related expenses (benefits), but these categories are too broad to permit Program staff to determine the amount of administrative costs. As a result, Program cannot measure service costs—As a result of its inability to measure administrative costs, the Program can-not determine the amount or percentage of Program monies that are spent on services such as helping smokers quit and prevent-ing youth from starting to use tobacco. There can be several lay-ers of entities charging administrative costs. Services are often provided by subcontractors, who may work under contract with other entities that hold contracts with county local projects. Thus, program services could account for much less than 90 percent of local project funding after the entities at each level (counties, contractors, and subcontractors) spend up to 10 percent of their budgets on administrative costs. 1 Kohler, Eric L. A Dictionary for Accountants. Prentice-Hall, Inc., 1975. There can be several layers of entities charging administrative costs. Finding III 27 OFFICE OF THE AUDITOR GENERAL Program Needs to Clarify Definitions and Establish Reporting Requirements In order to monitor compliance with the 10 percent administra-tive cost limit and maximize monies spent on tobacco control efforts, the Program needs to establish clear guidelines and re-porting requirements. In providing clearer guidelines, the Pro-gram also needs to differentiate administrative costs from indi-rect costs. This could be done as part of a thorough update to the Department’s accounting manual, which has not been revised since 1987. In setting more detailed reporting requirements, the Program needs to ensure that the reports will allow staff to as-sess whether each local project has kept its administrative costs within the 10 percent limit. In the future, once it has established a way to monitor compli-ance with the administrative cost limit, the Program should plan to measure the percentage of local project budgets that are used for program services. This will necessitate establishing proce-dures for tracking costs at every level down to the subcontractors who actually provide the prevention or cessation services. Only after such procedures are in place will the Program and its local projects be sufficiently accountable for their use of tobacco tax revenues. Finding III 28 OFFICE OF THE AUDITOR GENERAL Recommendations 1. The Department of Health Services and the Program should clarify the restriction on administrative costs by: a. Defining administrative costs, b. Differentiating between administrative costs and indirect costs, and c. Preparing guidelines for determining which specific costs should be included in the administrative cost category. 2. The Program should improve its monitoring of administra-tive costs by establishing reporting requirements for admin-istrative costs in its intergovernmental agreements. 3. The Department of Health Services should update its Ac-counting and Auditing Procedures Manual for ADHS Funded Programs, including adding the administrative cost definition and guidelines. 4. Once administrative costs are clearly defined and adminis-trative cost reporting requirements are in place, the Program should improve its accountability for its use of tobacco tax revenues by developing a plan for measuring the amount of local project funding used for providing program services. OFFICE OF THE AUDITOR GENERAL AGENCY RESPONSE Page 1 of 9 ADHS Response to the Performance Audit on the Arizona Tobacco Education and Prevention Program (TEPP) Overview The Arizona Department of Health Services (ADHS) agrees with the recommendations of the audit team. Overall, the report accurately reflects the challenges faced by the Arizona Tobacco Education and Prevention Program (TEPP) staff in establishing and administering a statewide, comprehensive tobacco control program. In reviewing the progress and problems with the TEPP in the control of tobacco use, it is useful to remember that Arizona is only the third state in the nation to establish such a program funded by a tobacco tax. This state is one of the pioneers in comprehensive tobacco control. As such, we have helped chart the way for others without the benefit of well-established and proven models on which to base a complex and sophisticated program attempting to help people make more healthful choices B whether trying to give adolescents the motivation and tools never to begin tobacco use or providing adult smokers with the motivation and means to stop smoking successfully. The model for a statewide, comprehensive tobacco control program is complicated, requiring integration of services among all the components, including media, local projects, and supporting services (such as the Arizona Smokers= Helpline, the Arizona Cessation Training and Evaluation Project, and the Arizona Tobacco Information Network). At the time the initiative was approved by voters, the tobacco prevention program within the ADHS consisted of only a half-time position funded primarily through preventive health block grant funds from the Centers for Disease Control and Prevention (CDC). The TEPP was established in early 1995. Within six months, an initial tobacco control plan for the state was developed and the key components of the program that is now TEPP were established. Tasks that were accomplished included hiring additional staff; the preparation, review and award of Requests for Proposals for media, local projects and the clearinghouse; and writing administrative rules. The county health departments have been key partners in developing an effective program response. Although only a few had any experience with tobacco control programs before the implementation of the statewide project, all have responded enthusiastically in learning about the needs and developing programs in new areas such as social marketing and aggressive media campaigns. Not surprisingly, many of the counties experienced the same type of growing pains that faced the TEPP during these early years. Staffing has continued to be a critical issue for TEPP, and has been a factor in addressing many of the findings cited in the Audit Report. For the first three years, the program operated with only five professional staff and two support staff. Key staffing positions have been added in the past two years, including a media liaison responsible for monitoring and administering the media contract, three health educators, and an evaluation coordinator. With this staff complement, we are addressing the issues raised in the Findings and Recommendations of this Audit Report. Page 2 of 9 Finding I - Improved Evaluations are Needed to Measure Program Effectiveness The TEPP has been candid in its reports to the Tobacco Use Prevention Advisory Committee (TUPAC) and to other external stakeholders about its deficiencies in evaluation. The TEPP actively recruited to fill the position of evaluation coordinator for nearly two years before finding a qualified individual, Pamela Goslar, Ph.D., who joined the TEPP staff in April 1999. Dr. Goslar has been working on a time line for the development of a statewide evaluation strategy (Recommendation 1). The first draft of this strategy was distributed and discussed with key stakeholders, including local projects (Recommendation 1a), on September 1, 1999. Information from this discussion will be incorporated into a final draft that will be presented to TUPAC in October. As part of her evaluation planning, Dr. Goslar has identified sources of information on evaluation relative to tobacco control (including the Needs Assessment Study done by the University of ArizonaBRecommendation 1b). These include the issues and considerations that must be incorporated into the strategy, the legislative evaluation requirements and programmatic guidelines for the program, and a summary of evaluation activities that are already underway. Also, in conjunction with the strategic planning process for evaluation, TEPP will also be developing its five-year strategic plan. The program has identified measurable goals and objectives that will serve as the basis for this planning. The TEPP will seek input from its key stakeholders, including local projects and TUPAC, as recommended by the audit report. With regard to measuring youth prevalence, in spite of the limitations in use of the results of the 1997 Statewide Youth Survey, we believe that other sources of data can be used to establish youth prevalence. The Arizona Criminal Justice Commission High School Substance Abuse Survey, which has collected information on Arizona youth tobacco use as far back as 1993, can be used to determine prevalence rates before the implementation of TEPP programs. The TEPP is working with the CDC on conducting the Youth Tobacco Survey (YTS), which has been developed by the CDC for use in all states. In order to be able to use some of the data collected by the 1997 Statewide Youth Survey, we will retain key questions and incorporate them into the YTS. It is TEPP=s intent to administer the YTS in schools next spring, and alternate years after that to establish a longitudinal measure of youth prevalence rates. However, because we are changing our data collection instrument and shifting from a telephone-based survey to a school-based survey, it will be at least three years before we can reasonably expect to have comparable data. Finding I Recommendations 1. The Program should develop an evaluation strategy, which includes a comprehensive evaluation plan that corresponds to Program goals. As part of its evaluation strategy, the Program should: a. Improve planning by seeking the input of national experts and other state officials Page 3 of 9 regarding ways to establish linkages between statewide and local efforts; b. Review and implement recommendations from the University of Arizona Cancer Center=s Needs Assessment Study to help refine Program activities; and c. Establish a youth smoking prevalence baseline. d. Continue efforts to work with the CDC on implementing the new Youth Tobacco Survey. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. 2. The Program should improve its communication with local projects by: a. Seeking recommendations for improvement from local project coordinators and evaluation staff; b. Providing feedback to local project personnel regarding evaluation reports and requirements; and c. Sharing evaluation information gathered by local projects to help improve local-level evaluations. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. 3. The Program should attempt to obtain tobacco-related information from other entities, such as: e. Tobacco consumption data from the Department of Revenue; f. Birth and death statistics from the Office of Vital Statistics; and g. Tobacco-related studies from the Disease Control Research Commission to help the Program meet its goals. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. 4. The Program should continue to coordinate surveys with other entities, particularly existing school-based surveys. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. Finding II - Program Needs to Improve Its Guidance of Tobacco Control Efforts The TEPP agrees with the audit report recommendation that written policies and procedures for all contractors are needed. In addition to a comprehensive policies and procedures manual, TEPP is developing a guidance manual for local projects to use in implementing prevention, education, cessation Page 4 of 9 services and in working toward tobacco-free environments as required by their contracts. We are striving in this to establish policies and procedures that both provide sufficient direction and allow local projects and their coalitions to create services based on individual community needs. We have given clear guidelines to local projects for school-based programs and for cessation. Local projects have been directed to utilize the CDC Guidelines for School-based Programs as well as the Agency for Health Care Policy and Research (AHCPR) Guidelines for Cessation. Although we made changes to the certification standards provided as part of the Arizona Cessation Training and Evaluation Project (ACTEV), we made many of those changes to reflect feedback received from early piloting of training programs. The TEPP acknowledges that we have had difficulty providing timely response and approval to local projects. We are committed to reducing the turnaround time. We have hired two health educators to work on local project administration. Their assignment is to assist in the review of local project reports and to provide feedback to local projects. We no longer have a six-month backlog of reports awaiting review, and the health educators have improved monitoring the compliance of local projects with monthly and quarterly reporting requirements. Communication between local projects and media contractor This finding cites examples of mis-communication primarily between local projects and the media contractor, Riester-Robb. The TEPP has aggressively taken steps to ensure improved communication, including the following: $ Filling the Media Coordinator Position: This TEPP staff position is dedicated to facilitating communication between the Media Contractor (Riester-Robb) and TEPP and monitoring communication between the Media Contractor and the TEPP funded projects (local projects, Arizona Tobacco Information Network, Arizona Smokers= Helpline). $ Designating a Local Projects Liaison within Riester-Robb: In 1997, the TEPP directed Riester- Robb to designate a staff person to be responsible for providing local projects with updates on developments related to the Statewide Media Campaign and to facilitate feedback from local projects on creative development, merchandise center products and event sponsorships. Local projects have indicated that this liaison person has improved their awareness of what is happening with the media campaign. $ Requiring each Local Project to Designate a Media Contact: Each local project has designated a key media contact, and the projects are aware that it is the responsibility of this person to ensure that information is shared with the project coordinator and other local project staff. In many cases, the local project coordinator is the media contact. Other local projects have a designated staff person or a subcontractor to handle media related issues. Page 5 of 9 Although the local project coordinators acknowledge that they still have some failures in internal communication within the local projects, the designation of a single media contact person has facilitated information being shared within the project. Similar problems were experienced with coordination of statewide media featuring the Arizona Smokers= Helpline. The Arizona Smokers= Helpline now receives updates on the media plan, which outlines when commercials will air. Riester-Robb has been directed to include Arizona Smokers= Helpline in its media updates, and Arizona Smokers= Helpline staff participate in key media creative and planning meetings. Role of the Arizona Tobacco Information Network (ATIN) and its relationship to local projects The audit report cites the development of Aduplicate materials@ as an issue. The role and function of the Arizona Tobacco Information Network (ATIN) is to collect tobacco-related education and prevention materials and make them available to local projects for their use. Local projects are also encouraged to develop their own Abrief intervention@ materials (such as brochures, handouts, fliers) as appropriate to address the specific needs of their communities. The ATIN does collect materials that are developed by individual local projects and makes them available to other projects that may find them appropriate for their use. The TEPP is encouraging ATIN to share only those project materials demonstrated to be effective and popular. To facilitate local project input, the ATIN conducts an annual needs assessment of local projects. This assessment allows the projects to focus their efforts in the upcoming contract year to suit their needs. In June 1999, ATIN presented TEPP with the results of a local projects customer-satisfaction survey that will serve as the basis for ongoing changes and improvements in ATIN services. As with the media contract, ATIN has a staff position designated to facilitate coordination with local projects. The TEPP and the ATIN are moving toward a more collaborative approach in the development of certain materials. One good example of that approach is the Spanish Language Materials Committee facilitated by ATIN. Local projects were asked to provide ATIN with Spanish language materials they had developed and/or were using in their communities. A group of staff from local projects was convened to review those materials and to identify gaps that needed to be filled. After a year=s work, we have a core of Spanish language materials that can be used by all local projects, including pieces that were adapted from materials developed by individual local projects. This model for materials development is being expanded to look at other needs, such as materials for Native Americans, African Americans and Asian Americans. In addition to the Spanish Language Materials Committee, ATIN has facilitated a review committee to look at the materials included in the ATIN catalog for local projects to order in bulk. Local projects are invited to participate in this activity regularly. Page 6 of 9 The ATIN has also developed uniform tobacco-free schools signage that will be distributed to local projects and school districts to assist in the implementation of new tobacco-free schools legislation. As TEPP moves toward promoting tobacco-free work sites, it will identify signage and materials currently being used by local projects and adapt them for use by all projects. Monitoring of TEPP funds by Arizona Department of Education (ADE) With regard to the use of TEPP funds by the Arizona Department of Education (ADE), TEPP has been working with the ADE on the use of tobacco education and prevention funding by school districts. Last year, TEPP directed ADE to dedicate these resources for the development and enforcement of tobacco-free school policies. There is specific language in the application going to school districts that directs them to work with local projects as a requirement of receipt of this funding. We have found it difficult to monitor compliance with these requirements, however. For the upcoming year, TEPP has again emphasized that these funds should be used for enforcement of tobacco-free school policies, and with the new legislative restriction prohibiting tobacco on school campuses and school-related events, districts have an opportunity to take advantage of these funds to implement the provisions of the law. Finally, the TEPP has taken steps to improve coordination of information among statewide and local project contractors. Coordinating communication and activities among the main components (local projects, ATIN, Arizona Smokers= Helpline, Statewide Media and ACTEV), is challenging, and TEPP has learned from experience that activities within one component invariably affect the work of the other three. To that end, we make significant efforts to ensure representation from each of the components affected by a specific issue in meetings and/or correspondence related to the topic. The TEPP is moving to the Awork group@ concept which brings these representatives together to discuss specific activities. The TEPP has also designated specific staff within each of the statewide projects to serve as a liaison to local projects to facilitate that communication. The audit report cites TEPP restriction of referral of calls from Arizona Smokers= Helpline to local project services. In the past, the TEPP did delay referral of callers until several key issues had been clarified. It was important to establish the ability of local projects to develop localized services, and more importantly, to handle and track those referrals. Many local projects were not prepared to accept referrals until this fiscal year, and TEPP has directed that statewide media (primarily television) would only be tagged with the Arizona Smokers= Helpline number, and that localized print and radio would be appropriate for local project tagging and referral. There were also logistical issues with Arizona Smokers= Helpline ability to refer callers efficiently to local projects, or to ensure that the information they had was accurate and current. The Arizona Smokers= Helpline is now capable of referring callers to individual local projects, which ensures that callers from outlying areas can be referred to local services. Page 7 of 9 Finding II Recommendations 1. The Program should conduct strategic planning with its advisory committee, statewide entities, and local projects, to include: a. Program goals and objectives; b. Allocation of resources; and c. Coordination of Program entities. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. 2. The Program should establish clear policies and procedures for statewide and local projects to define responsibilities, roles, and restrictions. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. 3. The Program should aggressively facilitate communication among statewide entities, local projects, and schools in order to share information and coordinate resources. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. 4. The Program should fully use the talents and experiences of its advisory committee and local projects staff to refine its activities. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. 5. The Legislature should consider whether to continue the Tobacco Use Prevention Advisory Committee (TUPAC). The finding of the Auditor General is agreed to and the audit recommendation will be implemented. Finding III - Program Does Not Monitor Administrative Costs The TEPP=s overall direction to its contractors is to ensure that the maximum amount of funding is dedicated to services, with limited administrative costs. To that end, TEPP has held contractors to a 10% limit for administrative costs. For the contracts beginning July 1, 1999, TEPP has clarified that policy to state more clearly that contracts are limited to 15% of Personnel and Employee Related Expenses. Page 8 of 9 While this limit on administrative costs applies to primary contracts, TEPP does not require, but recommends that contractors carry that policy forward to their subcontracts. The TEPP also does not distinguish between administrative costs and indirect costs. In preparing this response to the audit report (Recommendation 1a through 1c), the Division of Business and Financial Services (BFS) within ADHS has advised that the ADHS does not have a definition of administrative or indirect costs. The ADHS has advised the TEPP that it is reviewing this issue and will provide a definition of administrative and indirect costs and guidance for implementing the definition with the projects and their subcontractors. The Accounting and Auditing Procedures Manual for ADHS Funded Programs, is scheduled for a review (Recommendation 3). In the interim, the TEPP will consult with the ADHS Office of Auditing and develop broad distinctions between Aadministrative@ and Aservice@ costs, and has asked that they survey a sample of local project and statewide contracts through the subcontract level to determine if a disproportionate amount is being spent on administrative costs. This may assist in defining the problem with regard to the audit finding as well as in arriving at a definition of administrative and indirect costs. As soon as a definition has been established, TEPP will work with the appropriate representatives from the county health departments, the universities, other contractors, and representatives from ADHS to look at how such a definition will be applied to the TEPP contracts. With those guidelines in place, the TEPP can then proceed to identify an appropriate means for monitoring these costs among contractors on an ongoing basis, either through their current reporting requirement or through an independent audit (Recommendations 2 and 4). Finding III Recommendations 1. The Department of Health Services and the Program should clarify the restriction on administrative costs by: a. Defining administrative costs, b. Differentiating between administrative costs and indirect costs, and c. Preparing guidelines for determining which specific costs should be included in the administrative cost category. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. Page 9 of 9 2. The Program should improve its monitoring of administrative costs by establishing reporting requirements for administrative costs in its intergovernmental agreements. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. 3. The Department of Health Services should update its Accounting and Auditing Procedures Manual for ADHS Funded Programs, including adding the administrative cost definition and guidelines. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. 4. Once administrative costs are clearly defined and administrative cost reporting requirements are in place, the Program should improve its accountability for its use of tobacco tax revenues by developing a plan for measuring the amount of local project funding used for providing program services. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. Other Performance Audit Reports Issued Within the Last 12 Months 98-13 Private Enterprise Review Board 98-14 Adult Services 98-15 Podiatry Board 98-16 Board of Medical Examiners 98-17 Department of Health Services— Division of Assurance and Licensure 98-18 Governor’s Council on Develop-mental Disabilities 98-19 Personnel Board 98-20 Department of Liquor 98-21 Department of Insurance 98-22 State Compensation Fund 99-1 Department of Administration, Human Resources Division 99-2 Arizona Air Pollution Control Commission 99-3 Home Health Care Regulation 99-4 Adult Probation 99-5 Department of Gaming 99-6 Department of Health Services— Emergency Medical Services 99-7 Arizona Drug and Gang Policy Council 99-8 Department of Water Resources 99-9 Department of Health Services— Arizona State Hospital 99-10 Residential Utility Consumer Office/Residential Utility Consumer Board 99-11 Department of Economic Security— Child Support Enforcement 99-12 Department of Health Services— Division of Behavioral Health Services 99-13 Board of Psychologist Examiners 99-14 Arizona Council for the Hearing Impaired 99-15 Arizona Board of Dental Examiners 99-16 Department of Building and Fire Safety Future Performance Audit Reports Department of Health Services—Bureau of Epidemiology and Disease Control Services Department of Health Services—Division of Behavioral Health Services’ Sunset Factors Arizona State Board of Accountancy
Object Description
TITLE | Performance audit, Department of Health Services' Tobacco Education and Prevention Program |
CREATOR | Office of the Auditor General |
SUBJECT | Tobacco Education and Prevention Program; Arizona--Department of Health Services--Auditing; Tobacco habit--Arizona--Prevention |
Browse Topic |
Government and politics |
DESCRIPTION | This title contains one or more publications |
Language | English |
Publisher | Office of the Auditor General |
Material Collection | State Documents |
Acquisition Note | Report No. 99-17 |
Source Identifier | LG 6.2:R 36 |
Location | o42681109 |
REPOSITORY | Arizona State Library, Archives and Public Records--Law and Research Library |
Description
TITLE | Performance audit, Department of Health Services' Tobacco Education and Prevention Program |
DESCRIPTION | 50 pages (PDF version). File size: 311 KB |
TYPE |
Text |
RIGHTS MANAGEMENT | Copyright to this resource is held by the creating agency and is provided here for educational purposes only. It may not be downloaded, reproduced or distributed in any format without written permission of the creating agency. Any attempt to circumvent the access controls placed on this file is a violation of United States and international copyright laws, and is subject to criminal prosecution. |
DATE ORIGINAL | 1999-09 |
Time Period |
1990s (1990-1999) |
ORIGINAL FORMAT | Born Digital |
Source Identifier | LG 6.2:R 36 |
Location | o42681109 |
DIGITAL IDENTIFIER | 99-17.pdf |
DIGITAL FORMAT | PDF (Portable Document Format) |
REPOSITORY | Arizona State Library, Archives and Public Records--Law and Research Library. |
File Size | 318456 Bytes |
Full Text | State of Arizona Office of the Auditor General PERFORMANCE AUDIT Report to the Arizona Legislature By Debra K. Davenport Acting Auditor General September 1999 Report No. 99-17 DEPARTMENT OF HEALTH SERVICES' TOBACCO EDUCATION AND PREVENTION PROGRAM The Auditor General is appointed by the Joint Legislative Audit Committee, a bipartisan committee composed of five senators and five representatives. His mission is to provide independent and impar-tial information and specific recommendations to improve the operations of state and local government entities. To this end, he provides financial audits and accounting services to the state and political subdivisions and performance audits of state agencies and the programs they administer. The Joint Legislative Audit Committee Senator Tom Smith, Chairman Representative Roberta Voss, Vice-Chairman Senator Keith Bee Representative Robert Burns Senator Herb Guenther Representative Ken Cheuvront Senator Darden Hamilton Representative Andy Nichols Senator Pete Rios Representative Barry Wong Senator Brenda Burns Representative Jeff Groscost (ex-officio) (ex-officio) Audit Staff Shan Hays—Manager and Contact Person (602) 553-0333 Miriam Seymore—Audit Senior Angelica Gonzalez—Staff Raena Honan—Staff William Parker—Staff Copies of the Auditor General’s reports are free. You may request them by contacting us at: Office of the Auditor General 2910 N. 44th Street, Suite 410 Phoenix, AZ 85018 (602) 553-0333 Additionally, many of our reports can be found in electronic format at: www.auditorgen.state.az.us 2910 NORTH 44th STREET • SUITE 410 • PHOENIX, ARIZONA 85018 • (602) 553-0333 • FAX (602) 553-0051 DEBRA K. DAVENPORT, CPA ACTING AUDITOR GENERAL STATE OF ARIZONA OFFICE OF THE AUDITOR GENERAL September 13, 1999 Members of the Legislature The Honorable Jane Dee Hull, Governor Dr. James Allen, Director Department of Health Services Transmitted herewith is a report of the Auditor General, A Performance Audit of the Department of Health Services’ Tobacco Education and Prevention Program. This report is in response to a May 27, 1997, resolution of the Joint Legislative Audit Committee. This performance audit was conducted as part of the Sunset review set forth in A.R.S. §§41-2951 through 41-2957. This is the fifth in a series of reports to be issued on the Department of Health Services. This report addresses the Program’s need for improved evaluations to measure its effectiveness in preventing youth from starting to use tobacco, encouraging and assisting tobacco users to quit, and reducing exposure to secondhand smoke. We recommend that the Program develop an evaluation strategy and comprehensive plan that corresponds to Program goals, improve evaluation guidance given to the local projects it oversees, and coordinate with other entities to obtain useful tobacco-related information. In addition, the Program needs to improve its guidance of statewide and local projects in their tobacco control efforts. For example, it should develop written policies and procedures, facilitate communication among its statewide and local projects, and fully utilize its Tobacco Use and Prevention Advisory Committee. Further, the Program should monitor local projects’ administrative costs, which will require developing clear guidelines and establishing appropriate reporting requirements. Improved communication and coordination, along with the effective use of resources, will become increasingly important, as the Program’s future funding level represents a decrease of nearly 46 percent from the current level. September 13, 1999 Page –2- As outlined in its response, the Department agrees with all of our findings, and will imple-ment the recommendations. However, the decision to continue the Tobacco Use and Pre-vention Advisory Committee rests with the Legislature. My staff and I will be pleased to discuss or clarify items in the report. This report will be released to the public on September 14, 1999. Sincerely, Debbie Davenport Acting Auditor General Enclosure i OFFICE OF THE AUDITOR GENERAL SUMMARY The Office of the Auditor General has conducted a performance audit of the Department of Health Services’ Tobacco Education and Prevention Program (Program). This audit was conducted pursuant to a May 27, 1997, resolution of the Joint Legislative Audit Committee, and as part of the Sunset review set forth in Arizona Revised Statutes §§41-2951 through 41-2957. This is the fifth in a series of six audits of the Department of Health Services. The Program promotes a tobacco-free Arizona through work in three goal areas: preventing youth from starting to use tobacco, encouraging and assisting tobacco users to quit, and reducing exposure to secondhand smoke. With a staff of 16, the Program oversees tobacco control efforts on both local and statewide lev-els. On the local level, the Program has intergovernmental agreements with each county health department to provide to-bacco cessation counseling, conduct youth awareness programs, and establish community advisory committees. The Program also has intergovernmental agreements with various entities to support statewide tobacco control efforts and complement local activities. Statewide efforts include a tobacco control advertising campaign, the Arizona Smokers’ Helpline, materials and infor-mation clearinghouses, training for tobacco cessation counselors, and evaluations of Program performance. Improved Evaluations Are Needed to Measure Program Effectiveness (See pages 7 through 16) Evaluations at the state and local levels have not yet produced an adequate assessment of the Program’s tobacco control efforts. Arizona Laws 1995, Chapter 275 mandates evaluations to deter-mine whether the Program’s efforts are cost-effective and having any impact on tobacco use. Several of the Program’s attempted evaluations have encountered logistical and design problems. For example, the Program spent $900,000 on an evaluation that had to be terminated because it did not meet the Program’s The Program oversees to-bacco control efforts on both local and statewide levels. Summary ii OFFICE OF THE AUDITOR GENERAL information needs. In addition, the Program has been unable to establish a baseline on tobacco use among youth, which is an essential part of knowing the impact of its advertising campaign and other activities. The Program has had more success in meas-uring the prevalence of adult tobacco use, and expects to issue its first follow-up report on this topic by the end of 1999. More re-cent evaluations, such as a project that will provide a statewide assessment of cessation services by combining evaluation infor-mation from local projects, are more promising but have not yet been completed. Overall, the Program’s evaluation approach to date leaves it far short of knowing whether its programs are working. To improve its ability to evaluate the effect of its statewide and local activities, the Program needs to develop an evaluation strategy, improve its guidance to local project staff, and explore opportunities to coordinate its efforts with other entities that collect relevant data or conduct other studies of the Program’s target populations. The Program only recently hired a statewide evaluator, and as a result now has a better opportunity to make these improvements. The evaluation strategy should incorporate specific goals, objectives, and performance indicators, and it should help identify ways to better collect evaluation data. For example, since at least three other entities are studying youths’ attitudes and behaviors, some schools are declining to participate in school-based surveys. As an alternative, the Program can seek ways of using data collected by others. The strategy should also link statewide and local project efforts to the Program’s outcome objectives, in accordance with Centers for Disease Control guidelines (the federal agency responsible for coordinating to-bacco control efforts). Finally, in assembling its evaluation strat-egy, the Program should seek input from local project staff and incorporate recommendations from a 1998 assessment by the University of Arizona’s Cancer Center. Program Needs to Improve Its Guidance of Tobacco Control Efforts (See pages 17 through 24) The Program needs to improve the guidance and communica-tion it provides to local projects and statewide contractors. The Program has not developed manuals or written policies and The Program has yet to establish a baseline on youth tobacco use. Summary iii OFFICE OF THE AUDITOR GENERAL procedures, but instead waits for local project officials to seek guidance for many day-to-day decisions about their operations. The guidance provided is often inconsistent or untimely. For example, local project staff generally have to wait about three weeks to receive approval to put information about local area events on the statewide Internet Web site. By then, the events have already taken place. Similarly, feedback is often delayed for months on monthly reports submitted by local projects. The lack of clear and timely guidance and communication has resulted in duplicated efforts at events, redundant development of materials, and poor coordination between programs. For ex-ample, lack of communication between the statewide advertising program and the Smokers’ Helpline meant that the Helpline was unprepared to handle a heavy volume of calls when advertise-ments were broadcast that gave the Helpline’s 800 number. Likewise, the Program does not consistently inform local projects about its advertising contractor’s activities. Consequently, both the advertising contractor and local projects have sent crews to the same public events. To effectively guide Arizona’s tobacco control efforts, the Pro-gram needs to change the level and amount of guidance, making it more strategic and system-wide. The first step is to create a strategic plan that would help focus the Program’s efforts and better coordinate its statewide and local project resources. In the future, a strategic plan will become increasingly important, be-cause the Program’s current level of funding will decrease by nearly half as a buildup of reserves from the Program’s earliest years is rapidly spent. In developing a strategic plan and refo-cusing the guidance provided to its contractors, the Program should build on the efforts it has recently begun in seeking input fromits contractors and its advisory committee. Lack of coordination meant that the Helpline was unprepared to handle a heavy volume of calls. Summary iv OFFICE OF THE AUDITOR GENERAL Program Does Not Monitor Administrative Costs (See pages 25 through 28) Although the Program has established a limit of 10 percent on administrative costs, it cannot determine how much of the mon-ies provided for operating local education and prevention proj-ects are used for administration. It has not provided adequate guidelines for identifying such costs, and it has not required local projects to submit expenditure reports that contain enough detail to show which expenditures are administrative. As a result, the Program cannot determine the amount or percentage of Pro-gram monies that are spent on services, such as helping smokers quit and preventing youth from starting to use tobacco. Since services are often provided by several layers of entities charging administrative costs, Program services could account for much less that 90 percent of local project funding after counties, con-tractors, and subcontractors each spend up to 10 percent of their budgets on administrative costs. To ensure that local projects stay within the 10 percent limit, the Program needs to establish clear definitions and more complete reporting requirements. Several layers of entities charge administrative costs. v OFFICE OF THE AUDITOR GENERAL TABLE OF CONTENTS Page Introduction and Background ......................... 1 Finding I: Improved Evaluations Are Needed to Measure Program Effectiveness................................................ 7 Evaluation Requirements............................................... 7 Program Effectiveness Is Not Yet Known........................................................... 8 The Program Can Improve Statewide and Local Project Evaluation Efforts ........................................................... 11 Recommendations .......................................................... 16 Finding II: Program Needs To Improve Its Guidance of Tobacco Control Efforts .............................. 17 Program Structure .......................................................... 17 Program Does Not Provide Adequate Guidance or Communication to Statewide and Local Projects.......................................... 18 Several Factors Contribute to Ineffective Guidance....................................................... 21 Formal Planning and Written Policies Needed to Enhance Program Efficiency...................................... 23 Recommendations .......................................................... 24 Table of Contents vi OFFICE OF THE AUDITOR GENERAL TABLE OF CONTENTS (Concl’d) Page Finding III: Program Does Not Monitor Administrative Costs ................................... 25 The Program Has Not Adequately Defined or Monitored Administrative Costs ................................... 25 Program Needs to Clarify Definitions and Establish Reporting Requirements.................................................................. 27 Recommendations .......................................................... 28 Agency Response Table Table 1 Arizona Department of Health Services Tobacco Education and Prevention Program Statement of Revenues, Expenditures, and Other Changes in Fund Balance Years Ended June 30, 1997, 1998, and 1999 (Unaudited)................................................... 4 1 OFFICE OF THE AUDITOR GENERAL Item 1: Program Goals n Preventing youth from starting to use tobacco; n Encouraging and assisting tobacco users to quit; and n Reducing exposure to secondhand smoke INTRODUCTION AND BACKGROUND The Office of the Auditor General has conducted a performance audit of the Department of Health Services’ Tobacco Education and Prevention Program. This audit was conducted pursuant to a May 27, 1997, resolution of the Joint Legislative Audit Com-mittee, and as part of the Sunset review set forth in Arizona Re-vised Statutes §§41-2951 through 41-2957. This is the fifth in a series of six audits of the Department of Health Services. Tobacco Education and Prevention Program The Tobacco Education and Prevention Program (Program) originated in a 1994 voter initiative called the Tobacco Tax and Health Care Act. The Act increased the tobacco sales tax to 58 cents per pack and required that part of the revenue generated from the increase be used to provide funding for tobacco educa-tion and prevention. In 1995, to implement the Act, the Legisla-ture enacted House Bill 2275, which created the Health Educa-tion Account for tobacco prevention and reduction program and monies and directed the Department of Health Services to ad-minister the Account. The bill also established the Tobacco Use Prevention Advisory Committee, which is required to review Program activities and make recommendations to the Depart-ment. The legislation contains a provision that will terminate the Committee in December 1999. The Program seeks to promote a tobacco-free Arizona through a combination of statewide and local efforts. These efforts target three primary goals (see Item 1). To achieve these goals, the Program has adopted a comprehensive structure recommended by the federal Centers for Disease Control, which includes the following elements: Introduction and Background 2 OFFICE OF THE AUDITOR GENERAL n Local Projects—Intergovernmental agreements with each county health department support local efforts, such as to-bacco cessation counseling, youth awareness programs, and community partnerships established with local not-for-profit organizations. n Statewide Advertising Campaign—Under a competitive contract with an advertising company, the Program supports a highly visible, statewide tobacco control advertising cam-paign. The nationally recognized campaign employs a vari-ety of creative approaches including television and radio commercials, print advertisements, a merchandise center for campaign-related materials, and a mobile “Ash Kicker” vehi-cle with tobacco control displays aimed at youth. n Statewide Support Services—The Program provides funding and support for the Arizona Smokers’ Helpline, three information and materials resource clearinghouses, a Web site, and other services. These services are provided through intergovernmental agreements with Arizona’s three state universities. n Statewide Cessation and Evaluation Services—The Ari-zona Cessation Training and Evaluation Program (ACTEV), a Program-funded partnership between the University of Arizona and local projects, provides tobacco cessation certifi-cation and evaluation training for local projects and healthcare providers. The certification training focuses on skills required to encourage and assist tobacco users to quit and remain tobacco-free. The evaluation training teaches the skills necessary to evaluate local projects’ tobacco cessation activities. n Local and Statewide Enforcement Programs—The Pro-gram funds efforts to ensure compliance with laws restricting tobacco sales to minors. State and federal monies are allo-cated for enforcement. The Program receives grant monies through the federal Food and Drug Administration (FDA) enforcement program. The federal monies are administered by the Program and provided to communities for enforce-ment. Additionally, state-funded local projects in some areas, such as the City of Mesa, currently include enforcement in their activities. Introduction and Background 3 OFFICE OF THE AUDITOR GENERAL Budget, Personnel, and Organization The Program is primarily funded through state tobacco taxes with some additional funding from federal grants. The Program receives 23 percent of each dollar derived from the 58-cent-per-pack Arizona tobacco sales tax. The Program’s share of fiscal year 1998-99 tobacco tax revenue amounted to approximately $26.5 million. In addition, the Program has approximately $46 million in available carryover funds and interest earned, since the Legislature capped Program spending at $10 million and $15 million during its first and second years of operation, respec-tively. The Legislature later authorized an additional $9 million in available monies for the Program’s second year. (See Table 1, page 4.) Currently, the Legislature restricts Program spending to no more than 95 percent of the previous year’s tobacco tax reve-nues. For fiscal year 1998-99, the Program anticipates expenditures of approximately $32.7 million from state monies, including $1.25 million in transfers to the Arizona Department of Education for tobacco education in the public schools. The Program funds school-based tobacco control education efforts in accordance with federal Centers for Disease Control recommendations. Pro-gram expenditures primarily consist of services provided at the statewide and local projects level. A state Program office admin-isters the tobacco education and prevention projects as follows: The Program office, housed within the Division of Public Health Services, is authorized 16 staff. The staff are directed by an office chief and structured into major program areas, including local projects support, statewide support, financial management, marketing, and evaluations. Office administration expenditures for fiscal year 1998-99 were $726,000. Statewide functions are performed mainly through a contract with an advertising company and intergovernmental agree-ments with all three state universities. Expenditures for such services were approximately $21.4 million in fiscal year 1998-99, including $13.5 million for the advertising campaign contract. The Program anticipates expenditures of approxi-mately $32.7 million from state monies. Introduction and Background 4 OFFICE OF THE AUDITOR GENERAL Table 1 Arizona Department of Health Services Tobacco Education and Prevention Program Statement of Revenues, Expenditures, and Other Changes in Fund Balance Years Ended June 30, 1997, 1998, and 1999 (Unaudited) 1997 1998 1999 Revenues: Tobacco taxes $27,251,281 $26,779,642 $26,457,461 Interest on investments 2,145,792 2,745,938 2,777,685 Federal grants and contracts 48,501 81,962 264,964 Other 2,653 Total revenues 29,445,574 29,607,542 29,502,763 Expenditures: Personal services 402,929 393,543 461,315 Employee related 81,589 84,280 95,227 Professional and outside services 1 13,181,941 16,250,403 17,032,162 Travel, in-state 18,936 9,703 11,994 Travel, out-of-state 11,262 9,005 11,629 Aid to organizations 2 4,394,847 7,405,239 14,960,702 Other operating 107,725 50,200 112,442 Capital outlay 25,269 37,026 2,354 Allocated costs 10,360 21,074 30,898 Total expenditures 18,234,858 24,260,473 32,718,723 Excess of revenues over(under) expenditures 11,210,716 5,347,069 (3,215,960) Fund balance, beginning of year 32,437,568 43,648,284 48,995,353 Fund balance, end of year 3 $43,648,284 $48,995,353 $45,779,393 1 Includes $11.5, $13.2, and $13.5 million expended for media and sports contracts in 1997, 1998, and 1999, respectively. 2 Includes $3.1, $6.1, and $10.6 million passed through to counties for local projects in 1997, 1998, and 1999, respectively. In addition, includes $2.9 million passed through to Arizona universities in 1999 for statewide functions and $1.25 million passed through to the Arizona Department of Education for to-bacco education at the public schools in 1998 and 1999. 3 The Department has legislative guidelines for spending its fund balance. Source: The Arizona Financial Information System Revenues and Expenditures by Fund, Program, Organi-zation, and Object and Trial Balance by Fund reports; and Department-provided financial infor-mation for the years ended June 30, 1997, 1998, and 1999. Introduction and Background 5 OFFICE OF THE AUDITOR GENERAL Local projects funded by the Program through intergovernmen-tal agreements are operated by counties, tribal governments, and their subcontractors. These enti-ties were allocated about $13.7 million in fiscal year 1998-99 to develop activities aimed at get-ting communities, coalitions of concerned citizens, and interested groups involved in tobacco control goals and efforts. Maricopa County administers its local projects’ activities through subcontracts for programs and services while the remaining 14 counties directly administer the projects. Audit Scope and Methodology A combination of methods was used to study the issues ad-dressed in this audit. For example, Auditor General staff: n Reviewed the minutes from a sample of 8 Tobacco Use Pre-vention Advisory Committee meetings held from 1995 to 1998 and interviewed 7 of the 10 Committee members. n Reviewed contracts, intergovernmental agreements, and evaluations for statewide and local projects. Interviews were conducted with 15 members of local projects staff represent-ing all 15 Arizona counties, in addition to 7 local projects’ subcontractors. n Attended meetings of the Coalition for a Tobacco Free Ari-zona, the Program’s 4th annual conference, and regional meetings with the local projects held by the Program. n Interviewed representatives from all the organizations pro-viding statewide support services including the advertising campaign contractor, the Arizona Tobacco Information Net-work, the Arizona Prevention Resource Center, and the Ari-zona Cessation Training and Evaluation Program. Item 2: Tobacco Settlement In the future, Arizona expects to receive substantial funding from the multi-state tobacco industry settlement. However, the use of such funds has not been determined by the Gov-ernor and the Legislature. The future impact of settlement dollars on the Program fund-ing is unknown. Introduction and Background 6 OFFICE OF THE AUDITOR GENERAL n Conducted an extensive review of literature, including jour-nal articles, books, Internet Web sites, and reports from other states. n Interviewed administrators of similar programs in 7 other states.1 This audit contains findings and recommendations in three ar-eas: n The need to improve the Program’s ability to evaluate the success of statewide and local projects’ tobacco control ef-forts; n The need to improve the Program’s oversight of local proj-ects’ efforts; and n The Program’s inability to monitor local project administra-tive costs. This audit was conducted in accordance with government auditing standards. The Auditor General and staff express appreciation to the Di-rector of the Department of Health Services, the Assistant Direc-tor for the Division of Public Health Services, the management and staff of the Tobacco Education and Prevention Program, the management and staff of contracted statewide support services, and local project staff for their cooperation and assistance throughout the audit. 1 Other states contacted were California, Florida, Massachusetts, Min-nesota, Mississippi, Oregon, and Texas. These states were selected be-cause they had funding sources and goals similar to Arizona’s. 7 OFFICE OF THE AUDITOR GENERAL FINDING I IMPROVED EVALUATIONS ARE NEEDED TO MEASURE PROGRAM EFFECTIVENESS The Program has not adequately assessed the effectiveness of its tobacco prevention and cessation efforts. Improvements in statewide and local evaluations would result in better informa-tion about the Program’s success. The Program’s newness, to-gether with delays in hiring an evaluator to coordinate the vari-ous evaluation efforts, contributed to the slow start. An evaluator is now in place, giving the Program the opportunity to address past problems. To maximize the value of evaluation efforts, the Program should work with national experts and local projects to develop an evaluation strategy, including a plan to provide out-come measures and other data useful to state and local project staff. Evaluation Requirements Laws 1995, Chapter 275 requires biannual program evaluations to determine whether the Program’s education, prevention, and advertising campaigns are cost-effective and having an impact on tobacco use. Program evaluations should include the results of all monitoring of tobacco-related behaviors, attitudes, and health outcomes assessed at regular intervals. In following the Centers for Disease Control’s tobacco control program model, Arizona’s Program requires evaluations on both the statewide and local project levels. As part of their intergovernmental agreements with the Program, local projects are required to assist with data collection, including monthly and quarterly reports that may be monitored and accumulated for ongoing program evaluation. Local projects are also required to specify the evalua-tion budget and staffing they will use to conduct Program evaluations. Finding I 8 OFFICE OF THE AUDITOR GENERAL Program Effectiveness Is Not Yet Known While the Program has made some efforts to measure the suc-cess of statewide efforts over the last three years, its effectiveness has not been determined. Evaluation efforts since the Program’s 1995 inception have yielded inconclusive results; however, the Program is currently working on several evaluations that may provide more useful information in the future. Evaluation efforts to date have not determined effectiveness— The Program has attempted several evaluations, including a proc-ess evaluation that was expected to track both its short-term and long-term progress; an evaluation aimed at assessing the effec-tiveness of the statewide advertising campaign; and a survey intended to measure changes in tobacco use. However, logistical and design problems hampered the success of two of these ef-forts, and the third has not yet produced meaningful results. n Process Evaluation—The Program attempted a process evaluation of local project activities, but discontinued the ef-fort when the evaluation failed to provide the information needed. In 1996, the Program contracted with an organiza-tion for an evaluation it called Open Systems. The Open Systems effort was designed to examine the quantity and quality of activities performed by the local projects, and their progress toward Program goals. However, the evalua-tion’s design did not incorporate centralized planning. As a result, the design did not allow it to compare local projects to one another or to compare their effectiveness over time, and did not use random sampling or other methods in-tended to ensure valid results. Further, local project person-nel reported many problems with the evaluation, including the amount of time required by the process and frequent changes in the way they were instructed to implement the evaluation. Overall, the Program expended approximately $900,000 on a process evaluation that did not achieve its goals. n Youth Tobacco Survey—The Program did not obtain in-formation on youth tobacco use before starting its advertising campaign, and it struggled in subsequent efforts to measure youth’s use of and exposure to tobacco. The Arizona De- The Program expended approximately $900,000 before discontinuing a process evaluation that did not achieve its goals. Finding I 9 OFFICE OF THE AUDITOR GENERAL partment of Health Services’ Bureau of Public Statistics con-ducted a statewide telephone survey on youth smoking during 1996 and 1997, after the advertising campaign aimed at youth had begun. The survey attempted to estimate the prevalence of tobacco use, identify knowledge about access to tobacco products, and identify exposure to secondhand smoke. However, the survey’s timing undermines its poten-tial usefulness in measuring the Program’s impact, since the advertising campaign had already begun. In addition, the survey may have underestimated the prevalence rate of youth smoking, because it required parental consent and the youths may not have given honest answers in the presence of their parents. In order to determine the Program’s impact, a survey must be repeated to measure whether prevalence has changed from the first time the survey was administered. However, the Program is replacing the existing survey with another survey using a different methodology. While this will further delay the Program’s ability to measure its effectiveness, the new survey should produce more accurate information on youth tobacco use. n Advertising Evaluation—An evaluation of the statewide advertising campaign may yield more useful results, but key components will not be completed until September 1999. The first component of this advertising evaluation included a historical report to the Legislature, completed in October 1998, and a study of the cost-effectiveness of the advertising campaign, completed in January 1999. The October 1998 report incorporated a random sample telephone survey to measure exposure to and recognition of the 1996, 1997, and early 1998 advertisements and re-ported changes in attitudes and behaviors during the first years of the campaign. However, the January 1999 report could not fully answer questions about the cam-paign’s cost-effectiveness. According to Program offi-cials, Arizona was the first state that attempted to meas-ure cost-effectiveness of its advertising campaign. Using A.C. Neilsen (television) and Arbitron (radio) data, the study measured the number of people who saw or heard television, radio, billboard, and other advertisements. The study also incorporated results of adult and youth An evaluation of the statewide advertising campaign may yield more useful results. The Program still lacks a baseline on the prevalence of youth smoking. Finding I 10 OFFICE OF THE AUDITOR GENERAL smoking prevalence surveys to determine whether peo-ple remembered seeing the advertisements. It found, for example, that the cost per person in the adult population who saw television advertisements was $.01, and the cost per adolescent who remembered seeing the television advertisements was $25.66. However, the study was not designed to measure whether people had changed their smoking behaviors or attitudes. A yet-to-be-completed outcome component may yield more useful results. In order to look more closely at the link be-tween the advertising campaign and changes in attitudes and behaviors, the Program contracted with the Arizona Cancer Center at the University of Arizona, in 1998, to con-tinue evaluating the advertising campaign. Once completed, the evaluation’s outcome component will include six waves of surveys, including an in-depth survey of youth in schools, in-depth telephone interviews, and a random tele-phone sample of Hispanic households with women of childbearing age. The first wave of the outcome component was completed in late 1998. The fifth and sixth waves of surveys are due September 30, 1999, and were not com-pleted by the end of audit fieldwork. To date, the Program has expended approximately $265,000 for the advertising evaluation. Future evaluations may provide better information—In addition to the previously mentioned evaluations, the Program is cur-rently working on other evaluations that may provide more useful information in the future. However, it is too early to de-termine if these evaluations will yield desired outcomes. n Adult Tobacco Survey—The Program has had more suc-cess in its efforts to measure the prevalence of adult tobacco use than it has had with youth, although it has not yet measured any changes in prevalence of use since the Pro-gram’s inception. The initial survey, conducted in 1996 by the Department’s Bureau of Public Statistics, yielded infor-mation on smoking prevalence, individual tobacco use his-tories, and exposure to secondhand smoke. It also gathered information on reported tobacco control counseling pro-vided by health care professionals, and measured aware-ness of the Program’s tobacco control advertising campaign. The Program has had more success in its efforts to measure adult tobacco use. The cost-effectiveness study provided very limited results. Finding I 11 OFFICE OF THE AUDITOR GENERAL The Bureau of Public Statistics plans to repeat the survey and expects to release a report on the results in September 1999. This will be the first time that the Program can identify any changes in smoking prevalence since it began. n Project evaluations—Evaluation of several Program-funded projects should show whether or not those projects achieve their goals. For example, the Arizona Cessation Training and Evaluation Project (ACTEV) is a Program-funded partnership between the University of Arizona and local projects to help them set up and evaluate cessation services for tobacco-dependent community members. ACTEV will use a standardized questionnaire and provide evaluation feedback to local projects on the effectiveness of their cessation activities. ACTEV will also conduct a longitudinal study to track 500 people over time, a study to examine why people drop out of cessation activities, and a third study to at-tempt to gauge the power of brief interventions, using a random sample of the general population. Similarly, some other projects will include evaluations. CHAMPS, a tri-university partnership that trains youth to be “CHAMPS” by resisting tobacco, includes an evaluation compo-nent. In this project, participating youth will take tests at the beginning and end of the program to determine what knowl-edge they gained and whether they changed their attitudes or behavior. Finally, a pilot project for providing prevention coun-seling to Women, Infants, and Children (WIC) participants in four counties also includes an evaluation component to assess the intervention and its effect on the target population. The Program Can Improve Statewide and Local Project Evaluation Efforts While several factors have contributed to the Program’s inability to measure its effectiveness to date, the Program currently has the opportunity to address those factors in order to better assess its progress in the future. The Program recently addressed a Finding I 12 OFFICE OF THE AUDITOR GENERAL major deficiency in its evaluation structure by hiring a statewide evaluator. With the new evaluator in place, the Program should be able to develop an overall evaluation strategy and provide better direction and technical assistance to local projects. The Program should also explore opportunities to gain efficiencies by working with other entities to obtain needed information. Lack of evaluator, Program’s newness contributed to past evaluation problems—Delays in hiring a statewide evaluator contributed to the lack of continuity and focus for the Program’s evaluations. Until April 1999, the Program operated without a statewide evaluator. With the evaluator position now filled, Pro-gram officials anticipate improvements in both statewide and local project evaluations. Because officials lacked evaluation expertise and because na-tional state-operated tobacco control efforts were new, the Pro-gram made early mistakes that contributed to the difficulties encountered in evaluations to date. For example, eagerness to implement key parts of the Program, such as the advertising campaign, led directly to the Program’s inability to establish baseline information describing the prevalence of tobacco use among the campaign’s target populations before the advertise-ments began to appear. Similarly, inexperience with evaluation, and a lack of national models to learn from, made it difficult for management to predict that the process evaluation would not meet its needs. The Program should focus and refine its evaluation strategy— As a first step in improving its approach to evaluations, the Pro-gram should develop an evaluation plan to guide statewide and local efforts. Currently, the Program’s evaluation efforts lack continuity and focus, because no evaluation plan exists to help guide statewide or local project staff. A strategic plan is necessary to provide specific goals, objectives, and performance indicators, and it can also identify ways to better collect data. The new statewide evaluator intends to develop an evaluation plan by the end of 1999. In doing so, the evaluator should also consider ob-taining input from statewide, local project personnel, and other stakeholders in determining the future of Program evaluations. In developing this plan, the Program should consider the rec-ommendations made in a 1998 needs assessment conducted by The Program should develop an evaluation plan to guide statewide and local efforts. Finding I 13 OFFICE OF THE AUDITOR GENERAL the University of Arizona Cancer Center under a contract with the Program. This $25,000 study, specifically intended to identify ways to improve evaluations, created a profile of local projects’ current evaluation efforts and made numerous specific recom-mendations. However, the Program has not implemented the study’s recommendations for improving its overall evaluation strategy, such as forming a regional evaluation team, changing process evaluations, and improving local project efforts. Other study recommendations pertained to the kinds of evaluations the Program should conduct in the future, such as another proc-ess evaluation. The Program should improve its evaluation guidance to local projects—At the same time it improves its statewide evaluation plan, the Program needs to improve its oversight of local project evaluation efforts. Although local projects must include an evaluation component in their programs, the Program has not effectively communicated its evaluation expectations to local projects. Also, to maximize the usefulness of future local project evaluation efforts, the Program should link local project per-formance indicators to statewide goals. The Program needs to better communicate its evaluation expec-tations to local projects, as well as improve coordination between local project and statewide evaluation efforts. According to local project coordinators and evaluators, communication between the Program and local projects is inadequate. Local projects repeat-edly stated that the Program has not provided adequate evalua-tion guidance or technical assistance. Further, the Program does not have standardized evaluation practices that could allow local projects to work together or compare themselves to each other. Local projects want the new Program evaluator to provide more evaluation guidance and technical assistance by holding state-wide evaluation meetings and possibly creating an evaluation manual that would outline evaluation activities. Local projects also want the evaluator to provide feedback on whether their efforts are meeting statewide expectations and contributing to a reduction in tobacco use. According to the Centers for Disease Control, the federal agency that provides guidance to states in implementing a comprehen-sive approach to preventing and controlling tobacco use, pro-gram evaluation efforts should build upon and complement Communication between the Program and local projects is inadequate. Finding I 14 OFFICE OF THE AUDITOR GENERAL tobacco-related evaluation systems by linking statewide and local project efforts to outcome objectives. Currently, the Pro-gram has no structure in place that links performance indicators to statewide objectives. The Program could benefit from external opportunities and in-formation— Finally, after focusing its evaluation strategy and providing evaluation guidance, the Program can improve its efforts by working more collaboratively with related entities and developing and collecting better data. While the Program has attempted to coordinate evaluation efforts in the past, many opportunities to improve efficiencies still exist. Specifically, the Program can improve its evaluation efforts by working with other research entities, coordinating the administration of school-based surveys, and gathering available information about to-bacco. n Coordination with other entities—The Program could col-laborate with other entities, such as the Disease Control Re-search Commission, to obtain additional research related to tobacco. The Commission is a state agency whose purpose is to protect public health and safety by contracting with indi-viduals and organizations to conduct research into the cause and prevention of diseases, including behavioral changes such as smoking cessation. Currently, the Program’s Office Chief represents the Director of the Department of Health Services on the Commission. Because Commission funding derives from tobacco sales tax revenues, all Commission contractors are researching tobacco-related diseases in Ari-zona. Both the Program and the Commission could poten-tially benefit from collaborating on evaluations. For example, Massachusetts, another state with a program like Arizona’s, benefits from such research by contracting out small research projects to universities and other community researchers. Approximately five research projects per year are contracted out to fill in gaps created by other surveys or to follow up on problems or issues identified in certain populations. Recent projects include analyses of special population groups, in-cluding immigrants and elderly minorities. n Coordination on school-based surveys—According to local project coordinators and other evaluators, an excessive number of surveys are conducted in Arizona’s schools. As a Finding I 15 OFFICE OF THE AUDITOR GENERAL result, some school districts have declined to participate in school-based surveys. To reduce this saturation of surveys, the Program should continue to work with other entities that already obtain information about youth through school-based surveys, such as the Arizona Criminal Justice Commis-sion. The Program is currently coordinating with Behavioral Health Services on the High Risk Children’s Survey, which will include tobacco questions. n Other available information—The Program should work with other entities to gain a better understanding of tobacco use in Arizona. Other states’ tobacco programs compile in-formation gathered by various entities. For example, Oregon, California, and Massachusetts use their Departments of Revenue to collect tobacco consumption data from tobacco sales taxes. In addition, Oregon obtains birth and death in-formation from their vital statistics office to determine whether tobacco was used during a pregnancy or whether tobacco contributed to the death of a person. Oregon also uses data collected from the Oregon Office of Alcohol and Drug Abuse Program. Finding I 16 OFFICE OF THE AUDITOR GENERAL Recommendations 1. The Program should develop an evaluation strategy, which includes a comprehensive evaluation plan that corresponds to Program goals. As part of its evaluation strategy, the Pro-gram should: a. Improve planning by seeking the input of national experts and other state officials regarding ways to establish linkages be-tween statewide and local efforts; b. Review and implement recommendations from the University of Arizona Cancer Center’s Needs Assessment Study to help refine Program activities; c. Establish a youth smoking prevalence baseline; and d. Continue efforts to work with the CDC on implementing the new Youth Tobacco Survey. 2. The Program should improve its communication with local projects by: a. Seeking recommendations for improvement from local project coordinators and evaluation staff; b. Providing feedback to local project personnel regarding evaluation reports and requirements; and c. Sharing evaluation information gathered by local projects to help improve local-level evaluations. 3. The Program should attempt to obtain tobacco-related in-formation from other entities, such as: a. Tobacco consumption data from the Department of Revenue; b. Birth and death statistics from the Office of Vital Statistics; and c. Tobacco-related studies from the Disease Control Research Commission to help the Program meet its goals. 4. The Program should continue to coordinate surveys with other entities, particularly existing school-based surveys. 17 OFFICE OF THE AUDITOR GENERAL FINDING II PROGRAM NEEDS TO IMPROVE ITS GUIDANCE OF TOBACCO CONTROL EFFORTS The Program needs to improve its interactions with local projects and statewide entities. The Program does not provide adequate guidance or communication to its statewide and local projects, which limits its ability to effectively coordinate its resources. These problems stem from an approach that emphasizes cen-tralized approval and does not take advantage of general guid-ance that could be provided by the Program’s advisory commit-tee. Improvements are needed to increase efficiency, particularly since much less money will be available once the Program has spent the surplus funds that accumulated due to spending re-strictions during its first two years of operation. Program Structure The Tobacco Education and Prevention Program oversees and coordinates Arizona’s tobacco control campaign through local and statewide efforts. To operate tobacco control local projects, the Program has intergovernmental agreements with all Arizona county health departments. These local projects have four duties: n develop a local coalition of individuals and groups and write a plan to support the Program goals; n conduct prevention programs; n provide cessation services for tobacco users; and n promote tobacco-free schools, homes, and workplaces. Local projects vary in their sophistication. Since some local proj-ects began their efforts at different times, they differ in the num-ber and levels of services provided. For example, one of the first local projects, Pima County, began operations in 1996 and has staff and contractors working in all four Program areas. In Finding II 18 OFFICE OF THE AUDITOR GENERAL contrast, the La Paz local project was initiated in 1998 and has just begun to initiate tobacco control activities after spending its first year engaged in planning. In addition, during the first three years of operation, the Director of the Department of Health Services restricted Program efforts to prevention programs for youth and cessation services for pregnant women and their partners. However, some communities used other resources and initiated activities outside these restrictions. In addition to local project efforts, the Program also oversees contractors who conduct statewide tobacco control efforts. Statewide programs include an tobacco control advertising cam-paign, an tobacco control literature and materials clearinghouse, the Smokers’ Helpline, and cessation specialist training. State-wide efforts are intended to complement local project activities. . Program Does Not Provide Adequate Guidance or Communication to Statewide and Local Projects The Program has not adequately guided the tobacco control efforts of its statewide and local projects. While the Program has been in operation for four years, local projects lack basic operat-ing guidelines and timely assistance. Further, the lack of Pro-gram guidance and communication hinders the effective coordi-nation of resources among local projects and statewide entities. The Program does not provide needed guidance or adequate communication to local projects—The Program has not pro-vided consistent and timely technical assistance to statewide and local projects. Because tobacco education and prevention pro-grams are new, local projects rely on Program staff to provide guidance and clarify expectations. However, the Program has not met local projects’ needs with timely information and feed-back. Specific problems include: n No clear policies explaining expectations—The Program has not written policies and procedures or developed a man-ual for local projects to follow. As a result, local projects have received inconsistent instructions regarding Program opera-tions. For example, when the Program directed local projects to develop cessation programs for tobacco users who wished Local projects lack basic operating guidelines and timely assistance. Finding II 19 OFFICE OF THE AUDITOR GENERAL to quit, it had no established guidance for local projects to use in setting certification standards for cessation specialists. As part of the Arizona Cessation Training and Evaluation Proj-ect, a statewide effort to train cessation specialists was just getting underway. Local projects received changing and con-flicting instructions until certification standards were written several months after the training program began. n Untimely answers and approvals—The Program has been slow to respond to local projects’ requests for information and assistance. For example, local project staff cannot submit updated public information about local area events and community services through the statewide Internet Web site without Program office approval. However, approval rou-tinely takes three weeks, by which time the information is out of date. Similarly, it took the Program six months to approve an Internet mailing list and discussion forum that would help local projects communicate with each other. n Lack of feedback—The Program has not provided adequate feedback to its local projects. One valuable resource for feed-back is the monthly reporting from local projects submitted for Program staff review and comment. However, the Pro-gram usually has at least a six-month backlog of reports waiting for review, before feedback can be given to local projects. Most guidance given to local projects is conducted informally through e-mail and verbal updates that Program staff deliver at regional meetings, unsupported by written policies or guidelines. Lack of guidance and communication hinders effective coordina-tion of resources—The lack of Program guidance and communi-cation results in an ineffective use of resources. Problems created include duplication in staffing and materials; incomplete coordi-nation of tobacco control efforts with schools; and a lack of coor-dination among the Program’s advertising contractor, other statewide contractors, and local projects. n Duplicate staffing—The Program does not consistently inform local projects about its advertising contractor’s activi-ties. Consequently, both the advertising contractor and local projects have sent crews to the same public events. The lack of written poli-cies and procedures has led to inconsistent deci-sions. Finding II 20 OFFICE OF THE AUDITOR GENERAL n Duplicate materials—Both state and local projects are dupli-cating each other’s efforts in creating similar tobacco control materials. Tobacco prevention and education programs can-not function without accurate and professional public infor-mation. Although the Program contracts with Arizona State University to operate a statewide clearinghouse to create materials, some local projects are unaware of these efforts. The statewide clearinghouse is currently creating “quit kits” and three designs for decals that announce a smoke-free en-vironment. Since local projects are not regularly involved with the clearinghouse, they are creating their own “quit kits” and smoke-free decals. If the Program involved local projects in such clearinghouse activities, the local projects would not only be aware of clearinghouse efforts, they could participate in the development of materials suited to local needs. n The Program does not facilitate use of its monies in schools—The Program has not facilitated communication with schools regarding the use of tobacco control education monies. Specifically, the Program has not enforced the provi-sions of its interagency agreement with the Arizona Depart-ment of Education, which requires performance standards, monitoring, and regular reports on the use of Program to-bacco control monies provided to schools. Although school settings are crucial for tobacco use prevention, some schools are unaware that some of the monies they receive from ADE are Program monies to pay for tobacco control efforts. In ad-dition, the Program and ADE do not require schools receiv-ing these monies to work with statewide and local projects. As a result, schools have purchased curricula or attempted to develop it themselves, unaware they can obtain it without cost through the Program’s statewide clearinghouse. n The Program does not adequately coordinate statewide and local project activities—Crucial information is not shared between statewide programs, or between these pro-grams and local projects, making it more difficult for state-wide and local project staff to carry out their duties effec-tively. Experiences with the Smoker’s Helpline, a statewide program, provide an illustration of these problems. One set of problems involved the lack of coordination between Helpline and another statewide program, the advertising State and local projects are duplicating each other’s efforts. Crucial information is not shared. Finding II 21 OFFICE OF THE AUDITOR GENERAL campaign. The Helpline is staffed by counselors who fill re-quests for written material about tobacco use and advise smokers who wish to quit. The Helpline has not been notified in advance when its toll-free number is broadcast on radio and television stations throughout Arizona. Such notification would give Helpline managers an opportunity to prepare for the additional calls such announcements can bring. On sev-eral occasions after such announcements, the Helpline re-ceived hundreds of calls without enough counselors to han-dle them. Likewise, the Helpline has not been notified when Spanish-speaking commercials are aired, giving the staff no opportunity to add enough bilingual counselors. A second set of problems involved coordination between Helpline and local programs. The Program has not permitted the Helpline to refer callers to the local projects in their communities be-cause the Program has not decided if, or how, referrals should be made. As a result, callers may not be aware of Pro-gram services provided by their local project. Several Factors Contribute to Ineffective Guidance Two main factors have limited the Program’s ability to guide tobacco control efforts. First, Program management has overem-phasized supervision of contractors’ day-to-day operations at the expense of establishing broad policies and creating an effective network for open communication among its contractors. Second, the Program has not sought input from its advisory committee, which could help in developing such policies. Program has focused on supervising daily operations—Instead of developing clear guidelines for contractors to follow, the Pro-gram has required contractors to seek guidance and approval for all day-to-day operational decisions. Program management con-sidered such control necessary to establishing a new program, especially with inexperienced staff working at the local projects and to ensure that only appropriate activities received funding. The Program’s supervisory approach may also have contributed to its failure to obtain advice from local project staff on ways to improve the Program. Although Program officials occasionally Finding II 22 OFFICE OF THE AUDITOR GENERAL asked local project staff to fill out questionnaires, or to voice their concerns after hearing a presentation on Program decisions made by the officials, these efforts were inadequate for obtaining meaningful local project input. Recently, however, the Program has improved its efforts to communicate and to solicit input. In regional meetings held early in 1999, local project staff were asked to comment on a proposed form revision and to provide their views regarding how the Smokers’ Helpline might refer callers to the local projects. Limited input sought from advisory committee—The Program has not sought input from its advisory committee, which could assist it in developing broad policies to guide local projects and statewide contractors in their activities. The Tobacco Use Pre-vention Advisory Committee (TUPAC) is an advisory group of physicians, community leaders, and policy makers created by 1995 legislation. TUPAC is chaired by the director of the De-partment but has no decision-making authority. The Program does not consult them prior to making decisions, but TUPAC members are given information and may ask questions. During the audit, five of the public members of TUPAC were inter-viewed. Most report they do not interact with local projects and are unfamiliar with them. In addition, the majority of TUPAC’s public members expressed confusion over their role and some are reluctant to continue as members because they do not feel effective. The Legislature has recognized the need for stronger external oversight. In May 1999, the Legislature passed House Bill 2481. As part of this Department of Health Services’ omnibus bill, TUPAC would have been replaced by a Tobacco Revenue Use Spending and Tracking Commission. This legislation would have given the Commission many of the Program’s current oversight functions, such as developing a five-year plan for the use of monies in the Program’s Health Education Account, per-forming evaluations and reviews of programs and projects funded from the Account, and approving expenditures of mon-ies from the Account. However, this legislation was subse-quently vetoed by the Governor. Without new legislation, TUPAC will sunset on December 31, 1999. The Legislature has recognized the need for stronger external over-sight. Finding II 23 OFFICE OF THE AUDITOR GENERAL Formal Planning and Written Policies Needed to Enhance Program Efficiency In order to improve, the Program needs to change the level and amount of guidance it provides to statewide and local projects. The Program has lacked long-term strategic planning, and has not developed written policies and procedures. Advance plan-ning is essential to establish productive working relationships among statewide and local projects and to effectively coordinate resources. The Program needs to create a strategic plan for statewide and local projects. As noted earlier, lack of understanding of roles, responsibilities, and goals has created problems among local projects and statewide entities. A strategic plan could focus ef-forts of the Program and its statewide and local projects and help effectively coordinate these resources. As part of a strategic plan to guide tobacco control efforts and effectively coordinate re-sources, the Program should determine how to cope with de-creased availability of monies in the future. Currently, the Pro-gram is spending a previously unused surplus of approximately $43.6 million from the Health Education Account. By compari-son, Program management projects future tobacco tax revenues will generate approximately $20 million annually. The Program’s future funding level represents a decrease of nearly 46 percent of its current spending. (For more information on Program fund-ing, see Introduction and Background, pages 1 through 6.) The Program needs to develop written policies and procedures or manuals. Written guidelines would lessen the problem of local projects constantly seeking guidance from Program staff and management. Currently, Program guidelines are contained in separate documents and locations, including intergovern-mental agreements, annual renewal applications, and day-to-day e-mail responses to local project questions. The latter have been stored in the Program’s staff computers since 1995. Finding II 24 OFFICE OF THE AUDITOR GENERAL Recommendations 1. The Program should conduct strategic planning with its ad-visory committee, statewide entities, and local projects, to in-clude: a. Program goals and objectives; b. Allocation of resources; and c. Coordination of Program entities. 2. The Program should establish clear policies and procedures for statewide and local projects to define responsibilities, roles, and restrictions. 3. The Program should aggressively facilitate communication among statewide entities, local projects, and schools in order to share information and coordinate resources. 4. The Program should fully use the talents and experiences of its advisory committee and local projects staff to refine its ac-tivities. 5. The Legislature should consider whether to continue the Tobacco Use Prevention Advisory Committee (TUPAC). 25 OFFICE OF THE AUDITOR GENERAL FINDING III PROGRAM DOES NOT MONITOR ADMINISTRATIVE COSTS The Program cannot determine how much of the monies pro-vided for operating local education and prevention projects are used for administration. While its intergovernmental agreements restrict the amount of Program monies that can be spent on ad-ministrative costs, the Program cannot determine whether local projects adhere to the limit. In order to assess compliance with the limit and, eventually, measure the amount of local project funding spent on services, the Program needs to develop clear guidelines for determining administrative costs and establish appropriate reporting requirements. The Program Has Not Adequately Defined or Monitored Administrative Costs The Program has no assurance that administrative costs are within the limit it has established. The Program has not provided adequate guidelines for determining what should be considered as administrative costs and has not required local projects to submit detailed reports. As a result, the Program cannot deter-mine what portion of its monies has been spent on Program services. Program has established limit on administrative costs—The Program has established a limit on how much of its monies that local projects can spend on administrative costs. The Program's intergovernmental agreements restrict local projects to spending no more than 10 percent of budgeted monies on such costs. This is similar to state agencies, which typically spend 8 to 10 percent of their operating budgets on administrative costs. Program has not developed adequate guidelines for determining such costs—Despite setting the limit, the Program has not pro-vided clear guidelines for measuring administrative costs. The Program has not provided adequate guide-lines for determining administrative costs. Finding III 26 OFFICE OF THE AUDITOR GENERAL Its intergovernmental agreements instruct local projects to follow the standards and practices of the Department's 1987 accounting manual. However, this manual does not provide guidelines for administrative costs. Further, the Program uses the term “indi-rect costs” interchangeably with “administrative costs,” although the two terms have slightly different meanings. According to A Dictionary for Accountants,1 administrative costs are expenses incurred “in the general direction of an organization as a whole.” Indirect costs, according to the Department’s manual, include any costs incurred for a joint purpose and benefiting more than one program. This mix of terms further adds to the potential con-fusion about which costs should be included. Program does not require detailed reporting—Even if the Pro-gram provided clear guidelines regarding administrative costs, it could not determine whether local project administrative expen-ditures exceed the 10 percent limit because it does not require detailed tracking and reporting of costs. The Program requires financial reports showing total expenditures in categories such as personal services (salaries) and employee-related expenses (benefits), but these categories are too broad to permit Program staff to determine the amount of administrative costs. As a result, Program cannot measure service costs—As a result of its inability to measure administrative costs, the Program can-not determine the amount or percentage of Program monies that are spent on services such as helping smokers quit and prevent-ing youth from starting to use tobacco. There can be several lay-ers of entities charging administrative costs. Services are often provided by subcontractors, who may work under contract with other entities that hold contracts with county local projects. Thus, program services could account for much less than 90 percent of local project funding after the entities at each level (counties, contractors, and subcontractors) spend up to 10 percent of their budgets on administrative costs. 1 Kohler, Eric L. A Dictionary for Accountants. Prentice-Hall, Inc., 1975. There can be several layers of entities charging administrative costs. Finding III 27 OFFICE OF THE AUDITOR GENERAL Program Needs to Clarify Definitions and Establish Reporting Requirements In order to monitor compliance with the 10 percent administra-tive cost limit and maximize monies spent on tobacco control efforts, the Program needs to establish clear guidelines and re-porting requirements. In providing clearer guidelines, the Pro-gram also needs to differentiate administrative costs from indi-rect costs. This could be done as part of a thorough update to the Department’s accounting manual, which has not been revised since 1987. In setting more detailed reporting requirements, the Program needs to ensure that the reports will allow staff to as-sess whether each local project has kept its administrative costs within the 10 percent limit. In the future, once it has established a way to monitor compli-ance with the administrative cost limit, the Program should plan to measure the percentage of local project budgets that are used for program services. This will necessitate establishing proce-dures for tracking costs at every level down to the subcontractors who actually provide the prevention or cessation services. Only after such procedures are in place will the Program and its local projects be sufficiently accountable for their use of tobacco tax revenues. Finding III 28 OFFICE OF THE AUDITOR GENERAL Recommendations 1. The Department of Health Services and the Program should clarify the restriction on administrative costs by: a. Defining administrative costs, b. Differentiating between administrative costs and indirect costs, and c. Preparing guidelines for determining which specific costs should be included in the administrative cost category. 2. The Program should improve its monitoring of administra-tive costs by establishing reporting requirements for admin-istrative costs in its intergovernmental agreements. 3. The Department of Health Services should update its Ac-counting and Auditing Procedures Manual for ADHS Funded Programs, including adding the administrative cost definition and guidelines. 4. Once administrative costs are clearly defined and adminis-trative cost reporting requirements are in place, the Program should improve its accountability for its use of tobacco tax revenues by developing a plan for measuring the amount of local project funding used for providing program services. OFFICE OF THE AUDITOR GENERAL AGENCY RESPONSE Page 1 of 9 ADHS Response to the Performance Audit on the Arizona Tobacco Education and Prevention Program (TEPP) Overview The Arizona Department of Health Services (ADHS) agrees with the recommendations of the audit team. Overall, the report accurately reflects the challenges faced by the Arizona Tobacco Education and Prevention Program (TEPP) staff in establishing and administering a statewide, comprehensive tobacco control program. In reviewing the progress and problems with the TEPP in the control of tobacco use, it is useful to remember that Arizona is only the third state in the nation to establish such a program funded by a tobacco tax. This state is one of the pioneers in comprehensive tobacco control. As such, we have helped chart the way for others without the benefit of well-established and proven models on which to base a complex and sophisticated program attempting to help people make more healthful choices B whether trying to give adolescents the motivation and tools never to begin tobacco use or providing adult smokers with the motivation and means to stop smoking successfully. The model for a statewide, comprehensive tobacco control program is complicated, requiring integration of services among all the components, including media, local projects, and supporting services (such as the Arizona Smokers= Helpline, the Arizona Cessation Training and Evaluation Project, and the Arizona Tobacco Information Network). At the time the initiative was approved by voters, the tobacco prevention program within the ADHS consisted of only a half-time position funded primarily through preventive health block grant funds from the Centers for Disease Control and Prevention (CDC). The TEPP was established in early 1995. Within six months, an initial tobacco control plan for the state was developed and the key components of the program that is now TEPP were established. Tasks that were accomplished included hiring additional staff; the preparation, review and award of Requests for Proposals for media, local projects and the clearinghouse; and writing administrative rules. The county health departments have been key partners in developing an effective program response. Although only a few had any experience with tobacco control programs before the implementation of the statewide project, all have responded enthusiastically in learning about the needs and developing programs in new areas such as social marketing and aggressive media campaigns. Not surprisingly, many of the counties experienced the same type of growing pains that faced the TEPP during these early years. Staffing has continued to be a critical issue for TEPP, and has been a factor in addressing many of the findings cited in the Audit Report. For the first three years, the program operated with only five professional staff and two support staff. Key staffing positions have been added in the past two years, including a media liaison responsible for monitoring and administering the media contract, three health educators, and an evaluation coordinator. With this staff complement, we are addressing the issues raised in the Findings and Recommendations of this Audit Report. Page 2 of 9 Finding I - Improved Evaluations are Needed to Measure Program Effectiveness The TEPP has been candid in its reports to the Tobacco Use Prevention Advisory Committee (TUPAC) and to other external stakeholders about its deficiencies in evaluation. The TEPP actively recruited to fill the position of evaluation coordinator for nearly two years before finding a qualified individual, Pamela Goslar, Ph.D., who joined the TEPP staff in April 1999. Dr. Goslar has been working on a time line for the development of a statewide evaluation strategy (Recommendation 1). The first draft of this strategy was distributed and discussed with key stakeholders, including local projects (Recommendation 1a), on September 1, 1999. Information from this discussion will be incorporated into a final draft that will be presented to TUPAC in October. As part of her evaluation planning, Dr. Goslar has identified sources of information on evaluation relative to tobacco control (including the Needs Assessment Study done by the University of ArizonaBRecommendation 1b). These include the issues and considerations that must be incorporated into the strategy, the legislative evaluation requirements and programmatic guidelines for the program, and a summary of evaluation activities that are already underway. Also, in conjunction with the strategic planning process for evaluation, TEPP will also be developing its five-year strategic plan. The program has identified measurable goals and objectives that will serve as the basis for this planning. The TEPP will seek input from its key stakeholders, including local projects and TUPAC, as recommended by the audit report. With regard to measuring youth prevalence, in spite of the limitations in use of the results of the 1997 Statewide Youth Survey, we believe that other sources of data can be used to establish youth prevalence. The Arizona Criminal Justice Commission High School Substance Abuse Survey, which has collected information on Arizona youth tobacco use as far back as 1993, can be used to determine prevalence rates before the implementation of TEPP programs. The TEPP is working with the CDC on conducting the Youth Tobacco Survey (YTS), which has been developed by the CDC for use in all states. In order to be able to use some of the data collected by the 1997 Statewide Youth Survey, we will retain key questions and incorporate them into the YTS. It is TEPP=s intent to administer the YTS in schools next spring, and alternate years after that to establish a longitudinal measure of youth prevalence rates. However, because we are changing our data collection instrument and shifting from a telephone-based survey to a school-based survey, it will be at least three years before we can reasonably expect to have comparable data. Finding I Recommendations 1. The Program should develop an evaluation strategy, which includes a comprehensive evaluation plan that corresponds to Program goals. As part of its evaluation strategy, the Program should: a. Improve planning by seeking the input of national experts and other state officials Page 3 of 9 regarding ways to establish linkages between statewide and local efforts; b. Review and implement recommendations from the University of Arizona Cancer Center=s Needs Assessment Study to help refine Program activities; and c. Establish a youth smoking prevalence baseline. d. Continue efforts to work with the CDC on implementing the new Youth Tobacco Survey. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. 2. The Program should improve its communication with local projects by: a. Seeking recommendations for improvement from local project coordinators and evaluation staff; b. Providing feedback to local project personnel regarding evaluation reports and requirements; and c. Sharing evaluation information gathered by local projects to help improve local-level evaluations. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. 3. The Program should attempt to obtain tobacco-related information from other entities, such as: e. Tobacco consumption data from the Department of Revenue; f. Birth and death statistics from the Office of Vital Statistics; and g. Tobacco-related studies from the Disease Control Research Commission to help the Program meet its goals. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. 4. The Program should continue to coordinate surveys with other entities, particularly existing school-based surveys. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. Finding II - Program Needs to Improve Its Guidance of Tobacco Control Efforts The TEPP agrees with the audit report recommendation that written policies and procedures for all contractors are needed. In addition to a comprehensive policies and procedures manual, TEPP is developing a guidance manual for local projects to use in implementing prevention, education, cessation Page 4 of 9 services and in working toward tobacco-free environments as required by their contracts. We are striving in this to establish policies and procedures that both provide sufficient direction and allow local projects and their coalitions to create services based on individual community needs. We have given clear guidelines to local projects for school-based programs and for cessation. Local projects have been directed to utilize the CDC Guidelines for School-based Programs as well as the Agency for Health Care Policy and Research (AHCPR) Guidelines for Cessation. Although we made changes to the certification standards provided as part of the Arizona Cessation Training and Evaluation Project (ACTEV), we made many of those changes to reflect feedback received from early piloting of training programs. The TEPP acknowledges that we have had difficulty providing timely response and approval to local projects. We are committed to reducing the turnaround time. We have hired two health educators to work on local project administration. Their assignment is to assist in the review of local project reports and to provide feedback to local projects. We no longer have a six-month backlog of reports awaiting review, and the health educators have improved monitoring the compliance of local projects with monthly and quarterly reporting requirements. Communication between local projects and media contractor This finding cites examples of mis-communication primarily between local projects and the media contractor, Riester-Robb. The TEPP has aggressively taken steps to ensure improved communication, including the following: $ Filling the Media Coordinator Position: This TEPP staff position is dedicated to facilitating communication between the Media Contractor (Riester-Robb) and TEPP and monitoring communication between the Media Contractor and the TEPP funded projects (local projects, Arizona Tobacco Information Network, Arizona Smokers= Helpline). $ Designating a Local Projects Liaison within Riester-Robb: In 1997, the TEPP directed Riester- Robb to designate a staff person to be responsible for providing local projects with updates on developments related to the Statewide Media Campaign and to facilitate feedback from local projects on creative development, merchandise center products and event sponsorships. Local projects have indicated that this liaison person has improved their awareness of what is happening with the media campaign. $ Requiring each Local Project to Designate a Media Contact: Each local project has designated a key media contact, and the projects are aware that it is the responsibility of this person to ensure that information is shared with the project coordinator and other local project staff. In many cases, the local project coordinator is the media contact. Other local projects have a designated staff person or a subcontractor to handle media related issues. Page 5 of 9 Although the local project coordinators acknowledge that they still have some failures in internal communication within the local projects, the designation of a single media contact person has facilitated information being shared within the project. Similar problems were experienced with coordination of statewide media featuring the Arizona Smokers= Helpline. The Arizona Smokers= Helpline now receives updates on the media plan, which outlines when commercials will air. Riester-Robb has been directed to include Arizona Smokers= Helpline in its media updates, and Arizona Smokers= Helpline staff participate in key media creative and planning meetings. Role of the Arizona Tobacco Information Network (ATIN) and its relationship to local projects The audit report cites the development of Aduplicate materials@ as an issue. The role and function of the Arizona Tobacco Information Network (ATIN) is to collect tobacco-related education and prevention materials and make them available to local projects for their use. Local projects are also encouraged to develop their own Abrief intervention@ materials (such as brochures, handouts, fliers) as appropriate to address the specific needs of their communities. The ATIN does collect materials that are developed by individual local projects and makes them available to other projects that may find them appropriate for their use. The TEPP is encouraging ATIN to share only those project materials demonstrated to be effective and popular. To facilitate local project input, the ATIN conducts an annual needs assessment of local projects. This assessment allows the projects to focus their efforts in the upcoming contract year to suit their needs. In June 1999, ATIN presented TEPP with the results of a local projects customer-satisfaction survey that will serve as the basis for ongoing changes and improvements in ATIN services. As with the media contract, ATIN has a staff position designated to facilitate coordination with local projects. The TEPP and the ATIN are moving toward a more collaborative approach in the development of certain materials. One good example of that approach is the Spanish Language Materials Committee facilitated by ATIN. Local projects were asked to provide ATIN with Spanish language materials they had developed and/or were using in their communities. A group of staff from local projects was convened to review those materials and to identify gaps that needed to be filled. After a year=s work, we have a core of Spanish language materials that can be used by all local projects, including pieces that were adapted from materials developed by individual local projects. This model for materials development is being expanded to look at other needs, such as materials for Native Americans, African Americans and Asian Americans. In addition to the Spanish Language Materials Committee, ATIN has facilitated a review committee to look at the materials included in the ATIN catalog for local projects to order in bulk. Local projects are invited to participate in this activity regularly. Page 6 of 9 The ATIN has also developed uniform tobacco-free schools signage that will be distributed to local projects and school districts to assist in the implementation of new tobacco-free schools legislation. As TEPP moves toward promoting tobacco-free work sites, it will identify signage and materials currently being used by local projects and adapt them for use by all projects. Monitoring of TEPP funds by Arizona Department of Education (ADE) With regard to the use of TEPP funds by the Arizona Department of Education (ADE), TEPP has been working with the ADE on the use of tobacco education and prevention funding by school districts. Last year, TEPP directed ADE to dedicate these resources for the development and enforcement of tobacco-free school policies. There is specific language in the application going to school districts that directs them to work with local projects as a requirement of receipt of this funding. We have found it difficult to monitor compliance with these requirements, however. For the upcoming year, TEPP has again emphasized that these funds should be used for enforcement of tobacco-free school policies, and with the new legislative restriction prohibiting tobacco on school campuses and school-related events, districts have an opportunity to take advantage of these funds to implement the provisions of the law. Finally, the TEPP has taken steps to improve coordination of information among statewide and local project contractors. Coordinating communication and activities among the main components (local projects, ATIN, Arizona Smokers= Helpline, Statewide Media and ACTEV), is challenging, and TEPP has learned from experience that activities within one component invariably affect the work of the other three. To that end, we make significant efforts to ensure representation from each of the components affected by a specific issue in meetings and/or correspondence related to the topic. The TEPP is moving to the Awork group@ concept which brings these representatives together to discuss specific activities. The TEPP has also designated specific staff within each of the statewide projects to serve as a liaison to local projects to facilitate that communication. The audit report cites TEPP restriction of referral of calls from Arizona Smokers= Helpline to local project services. In the past, the TEPP did delay referral of callers until several key issues had been clarified. It was important to establish the ability of local projects to develop localized services, and more importantly, to handle and track those referrals. Many local projects were not prepared to accept referrals until this fiscal year, and TEPP has directed that statewide media (primarily television) would only be tagged with the Arizona Smokers= Helpline number, and that localized print and radio would be appropriate for local project tagging and referral. There were also logistical issues with Arizona Smokers= Helpline ability to refer callers efficiently to local projects, or to ensure that the information they had was accurate and current. The Arizona Smokers= Helpline is now capable of referring callers to individual local projects, which ensures that callers from outlying areas can be referred to local services. Page 7 of 9 Finding II Recommendations 1. The Program should conduct strategic planning with its advisory committee, statewide entities, and local projects, to include: a. Program goals and objectives; b. Allocation of resources; and c. Coordination of Program entities. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. 2. The Program should establish clear policies and procedures for statewide and local projects to define responsibilities, roles, and restrictions. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. 3. The Program should aggressively facilitate communication among statewide entities, local projects, and schools in order to share information and coordinate resources. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. 4. The Program should fully use the talents and experiences of its advisory committee and local projects staff to refine its activities. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. 5. The Legislature should consider whether to continue the Tobacco Use Prevention Advisory Committee (TUPAC). The finding of the Auditor General is agreed to and the audit recommendation will be implemented. Finding III - Program Does Not Monitor Administrative Costs The TEPP=s overall direction to its contractors is to ensure that the maximum amount of funding is dedicated to services, with limited administrative costs. To that end, TEPP has held contractors to a 10% limit for administrative costs. For the contracts beginning July 1, 1999, TEPP has clarified that policy to state more clearly that contracts are limited to 15% of Personnel and Employee Related Expenses. Page 8 of 9 While this limit on administrative costs applies to primary contracts, TEPP does not require, but recommends that contractors carry that policy forward to their subcontracts. The TEPP also does not distinguish between administrative costs and indirect costs. In preparing this response to the audit report (Recommendation 1a through 1c), the Division of Business and Financial Services (BFS) within ADHS has advised that the ADHS does not have a definition of administrative or indirect costs. The ADHS has advised the TEPP that it is reviewing this issue and will provide a definition of administrative and indirect costs and guidance for implementing the definition with the projects and their subcontractors. The Accounting and Auditing Procedures Manual for ADHS Funded Programs, is scheduled for a review (Recommendation 3). In the interim, the TEPP will consult with the ADHS Office of Auditing and develop broad distinctions between Aadministrative@ and Aservice@ costs, and has asked that they survey a sample of local project and statewide contracts through the subcontract level to determine if a disproportionate amount is being spent on administrative costs. This may assist in defining the problem with regard to the audit finding as well as in arriving at a definition of administrative and indirect costs. As soon as a definition has been established, TEPP will work with the appropriate representatives from the county health departments, the universities, other contractors, and representatives from ADHS to look at how such a definition will be applied to the TEPP contracts. With those guidelines in place, the TEPP can then proceed to identify an appropriate means for monitoring these costs among contractors on an ongoing basis, either through their current reporting requirement or through an independent audit (Recommendations 2 and 4). Finding III Recommendations 1. The Department of Health Services and the Program should clarify the restriction on administrative costs by: a. Defining administrative costs, b. Differentiating between administrative costs and indirect costs, and c. Preparing guidelines for determining which specific costs should be included in the administrative cost category. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. Page 9 of 9 2. The Program should improve its monitoring of administrative costs by establishing reporting requirements for administrative costs in its intergovernmental agreements. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. 3. The Department of Health Services should update its Accounting and Auditing Procedures Manual for ADHS Funded Programs, including adding the administrative cost definition and guidelines. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. 4. Once administrative costs are clearly defined and administrative cost reporting requirements are in place, the Program should improve its accountability for its use of tobacco tax revenues by developing a plan for measuring the amount of local project funding used for providing program services. The finding of the Auditor General is agreed to and the audit recommendation will be implemented. Other Performance Audit Reports Issued Within the Last 12 Months 98-13 Private Enterprise Review Board 98-14 Adult Services 98-15 Podiatry Board 98-16 Board of Medical Examiners 98-17 Department of Health Services— Division of Assurance and Licensure 98-18 Governor’s Council on Develop-mental Disabilities 98-19 Personnel Board 98-20 Department of Liquor 98-21 Department of Insurance 98-22 State Compensation Fund 99-1 Department of Administration, Human Resources Division 99-2 Arizona Air Pollution Control Commission 99-3 Home Health Care Regulation 99-4 Adult Probation 99-5 Department of Gaming 99-6 Department of Health Services— Emergency Medical Services 99-7 Arizona Drug and Gang Policy Council 99-8 Department of Water Resources 99-9 Department of Health Services— Arizona State Hospital 99-10 Residential Utility Consumer Office/Residential Utility Consumer Board 99-11 Department of Economic Security— Child Support Enforcement 99-12 Department of Health Services— Division of Behavioral Health Services 99-13 Board of Psychologist Examiners 99-14 Arizona Council for the Hearing Impaired 99-15 Arizona Board of Dental Examiners 99-16 Department of Building and Fire Safety Future Performance Audit Reports Department of Health Services—Bureau of Epidemiology and Disease Control Services Department of Health Services—Division of Behavioral Health Services’ Sunset Factors Arizona State Board of Accountancy |