Annual evaluation, Healthy Families Program |
Previous | 1 of 3 | Next |
|
|
Small
Medium
Large
Extra Large
Full-size
Full-size archival image
|
This page
All
|
State of Arizona Office of the Auditor General PERFORMANCE AUDIT Report to the Arizona Legislature By Debra K. Davenport Auditor General HEALTHY FAMILIES PROGRAM February 2001 Report No. 01-02 The Auditor General is appointed by the Joint Legislative Audit Committee, a bipartisan committee composed of five senators and five representatives. Her mission is to provide independent and impar-tial information and specific recommendations to improve the operations of state and local government entities. To this end, she provides financial audits and accounting services to the state and political subdivisions and performance audits of state agencies and the programs they administer. Audit Staff Carol Cullen—Manager and Contact Person (602) 553-0333 Joanne Dukeshire—Team Leader Laurie Cohen—Team Member Copies of the Auditor General’s reports are free. You may request them by contacting us at: Office of the Auditor General 2910 N. 44th Street, Suite 410 Phoenix, AZ 85018 (602) 553-0333 Additionally, many of our reports can be found in electronic format at: www.auditorgen.state.az.us 2910 NORTH 44th STREET • SUITE 410 • PHOENIX, ARIZONA 85018 • (602) 553-0333 • FAX (602) 553-0051 DEBRA K. DAVENPORT, CPA AUDITOR GENERAL STATE OF ARIZONA OFFICE OF THE AUDITOR GENERAL February 15, 2001 Members of the Arizona Legislature The Honorable Jane Dee Hull, Governor Mr. John Clayton, Director Department of Economic Security Transmitted herewith is a report of the Auditor General, an evaluation of the Healthy families Program. This evaluation was conducted pursuant to A.R.S. §41-1279.08. I am also transmitting with this report a copy of the Report Highlights for this evaluation to provide a quick summary for your convenience. As outlined in its response, the Department agrees with all of the findings and recommendations. My staff and I will be pleased to discuss or clarify items in the report. This report will be released to the public on February 16, 2001. Sincerely, Debbie Davenport Auditor General Enclosure i OFFICE OF THE AUDITOR GENERAL SUMMARY The Office of the Auditor General has completed an evaluation of the Healthy Families Program. The Office of the Auditor Gen-eral is required pursuant to Arizona Revised Statutes (A.R.S.) §41-1279.08 to evaluate the Healthy Families Program annually. This report contains information related to program procedures and effectiveness. Healthy Families—Arizona is a child abuse prevention and child wellness and development program administered by the De-partment of Economic Security (DES). The Healthy Families Program has five goals: (1) reduce child abuse and neglect; (2) promote child wellness and proper development; (3) strengthen family relations; (4) promote family unity; and (5) reduce de-pendency on drugs and alcohol. The program coordinates with hospitals to identify mothers giving birth in specific geographic areas whose family character-istics may place them at risk for committing child abuse or ne-glect, and/or whose babies may face increased health risk. Healthy Families provides services through various contractors, which include medical centers, local health departments, and social service agencies. These contractors provide services at 23 sites in 10 counties. The program is based on a home-visitation model. Home visitors regularly visit families to offer support, educational materials, and referrals to needed resources. Home visitors also help fami-lies develop family support plans and encourage positive parent-child interaction to strengthen bonds and promote development. Participation in the program is voluntary and may continue for up to five years. In fiscal year 2000, 1,254 families were enrolled in the program. Summary ii OFFICE OF THE AUDITOR GENERAL Healthy Families Participant Child Abuse Rates Do Not Differ Significantly from the Comparison Group (See pages 13 through 19) For the second year in a row, comparison of child abuse rates for Healthy Families participants and a comparison group of fami-lies who left the program shows that abuse rates were not sig-nificantly different. Specifically, 1.7 percent of participant fami-lies received a substantiated report of abuse and 1.8 percent of families who left the program before receiving four home visits received a substantiated report. These rates are comparable to the rates reported in the previous Office of the Auditor General evaluation of the program (see Report No. 00-1), in which 1.6 percent of participants and 1.4 percent of the comparison group received substantiated reports of abuse. While one possible explanation for these findings might be that the program does not have a significant effect on child abuse, other explanations may also account for the similarity in out-comes between the two groups. First, it is possible that the effects of the program on preventing child abuse cannot be adequately measured by the current evaluation design because of changes in DES procedures for substantiating child abuse reports. In par-ticular, the occurrence of substantiated abuse reports has de-creased since the implementation of an appeals process. Second, preliminary analysis of when abuse occurs for both program participants and the comparison group suggests that abuse is likely to occur a year or more after enrollment, and in most cases, after the family has left the program. Thus, participants may not be enrolled in the program at the time when the risk of child abuse increases. Healthy Families Program Provides Health Referrals, Encourages On-Time Immunizations, and Promotes Safety (See pages 21 through 26) The Healthy Families Program has been successful in providing health referrals, encouraging parents to get their children immu-nized, and in promoting home safety. On average, the majority Summary iii OFFICE OF THE AUDITOR GENERAL of children in the program are fully immunized and most chil-dren have a primary health care provider. Additionally, Healthy Families staff regularly conduct assessments of the physical and social development of children enrolled in the program. Parents improved their compliance with home safety measures such as covering electrical outlets and keeping poisons and choking hazards out of reach. Parenting Stress Measure Shows Improved Family Relations (See pages 27 through 29) Healthy Families parents showed significant decreases in indica-tors of parental stress after one year in the program, suggesting that program services may improve parent-child relations. In addition, parents with higher stress levels related to areas such as social isolation and depression also received referrals to mental health services at a higher rate than parents with lower stress. To assist families in improving parent-child interaction, home visi-tors regularly provide emotional support and information to help parents understand their child’s behavior. Other Pertinent Information (See pages 31 through 34) The previous report issued by the Office of the Auditor General (see Report No. 00-1) offered several recommendations to im-prove the measurement of the program’s effect on family func-tioning and reduction of substance abuse. The previous evalua-tion also recommends tracking referrals for children with devel-opmental delays. To address these recommendations, the pro-gram discontinued using the Home Observation for Measure-ment of the Environment (HOME) questionnaire and will in-stead use the Parenting Stress Index (PSI) to measure the pro-gram’s effect on family relations. The program also replaced the CAGE substance abuse screen with a new questionnaire that can measure the program’s effect on reducing drug/alcohol depend-ency over time. Finally, the program added questions to its fam-ily update form, which will enable staff to track whether children with developmental delays are offered the appropriate referrals. Summary iv OFFICE OF THE AUDITOR GENERAL Statutory Annual Evaluation Components (See pages 35 through 44) This report also contains information that addresses various components of the Healthy Families evaluation statute. These components include information regarding participant demo-graphics such as mother’s age, marital status, income, ethnicity, and household size. Other areas covered in the statutory compo-nents section contain information about enrollment and disen-rollment and employment. v OFFICE OF THE AUDITOR GENERAL TABLE OF CONTENTS Page Introduction and Background.......................... 1 Finding I: Healthy Families Participant Child Abuse Rates Do Not Differ Significantly from the Comparison Group................................ 13 Child Abuse Rates for Healthy Families Participants and Comparison Group Similar............................................ 13 Possible Explanations of Results......................................................................... 16 Finding II: Healthy Families Program Provides Health Referrals, Encourages On-Time Immunizations, and Promotes Safety ............................................................ 21 Background..................................................................... 21 Participants Receive Developmental Screenings, Referrals, and On-Time Immunizations................................................................ 22 Participants Show Significant Improvements in Home Safety...................................... 24 Recommendations .......................................................... 26 Finding III: Parenting Stress Measure Shows Improved Family Relations............. 27 Background..................................................................... 27 Participants’ Stress Reduced After 12 Months.............................................................. 27 Table of Contents vi OFFICE OF THE AUDITOR GENERAL TABLE OF CONTENTS (Cont’d) Page Finding III: (Concl’d) Parents with Higher Stress More Likely to Receive Mental Health Referrals .............................................................. 29 Home Visitors Provide Information on Parent-Child Bonding..................................................... 29 Other Pertinent Information ............................. 31 Statutory Annual Evaluation Components...... 35 Appendix ........................................................... a-i Agency Response Figures Figure 1 Healthy Families Program Contract Provider Locations Year Ended June 30, 2000............................. 8 Figure 2 Healthy Families Program Number of Participants and Comparison Group Members by County January 1994 through May 2000.................. 37 Figure 3 Healthy Families Program Reasons for Exit January 1994 through May 2000.................. 40 Figure 4 Healthy Families Program Employment Rates of Mothers and Fathers in Program January 1994 through May 2000................. 44 Table of Contents vii OFFICE OF THE AUDITOR GENERAL TABLE OF CONTENTS (Cont’d) Page Tables Table 1 Healthy Families Program Schedule of Distributions by Funding Source and Contractor, and Average State Contribution per Family Year Ended June 30, 2000 (Unaudited)................................................... 7 Table 2 Healthy Families Program Percentage of Participants and Comparison Group Members with Substantiated and Unsubstantiated Reports of Abuse and Referrals to Family Builders Reports Received Between January 1, 1998 and May 31, 2000................ 15 Table 3 Healthy Families Program Percentage of Participants and Comparison Group Members with Substantiated Abuse Reports Office of the Auditor General Evaluations.................................................... 16 Table 4 Healthy Families Program Ages and Stages Questionnaire Percentage of Children Developing at Age-Appropriate Levels January 1994 to May 2000............................. 24 Table 5 Healthy Families Program Percentage of Safety Measures Implemented by Age of Child July 1998 to May 2000................................... 25 Table 6 Healthy Families Program Average Parenting Stress Scores After 3 Weeks and 12 Months in the Program July 1998 to May 2000................................... 28 Table of Contents viii OFFICE OF THE AUDITOR GENERAL TABLE OF CONTENTS (Concl’d) Page Tables (Concl’d) Table 7 Healthy Families Program Participant and Comparison Group Member Profile January 1994 through May 2000.................. 36 Table 8 Healthy Families Program Participants and Comparison Group Members Profile of Family Characteristics January 1994 through May 2000.................. 38 Table 9 Healthy Families Program Months of Participation for Families at Exit January 1994 through May 2000.................. 40 1 OFFICE OF THE AUDITOR GENERAL INTRODUCTION AND BACKGROUND The Office of the Auditor General has completed an evaluation of the Healthy Families Program. According to Arizona Revised Statutes (A.R.S.) §41-1279.08, the Office of the Auditor General is required to evaluate the Healthy Families Program annually. This report contains information regarding the program's proce-dures and outcomes. Healthy Families Offers a Preventative Approach to the Problem of Child Abuse Healthy Families—Arizona is a child abuse prevention and child wellness and development program administered by the De-partment of Economic Security (DES). Preventing child abuse is a serious concern in Arizona, as in other states. During fiscal year 1999, Arizona's Department of Economic Security, Division of Child Protective Services received 32,631 reports of maltreatment toward children. Twenty-five percent of these reports were sub-sequently substantiated.1 The State Legislature established Healthy Families—Arizona as a pilot program in 1994, and gave it permanent program status in 1998. Healthy Families attempts to identify families at risk for committing child abuse or neglect and that may have poor health outcomes. The program provides participants with sup-port, education about child development and referrals for nutri-tion, medical care, counseling, and education and employment programs. Arizona's program is part of a national initiative pro-moted by Prevent Child Abuse America. It is currently offered in 39 states, the District of Columbia, and Canada and serves over 400 communities. Program enrollment and participation are 1 Department of Economic Security. Child Welfare Reporting Requirements: Annual Report for the Period of July 1, 1998 through June 30, 1999. Substanti-ated reports are those reports DES determines met the standard of prob-able cause to conclude that abuse occurred. Note: Of the total reports for fiscal year 1999, 153 fell within the jurisdiction of military or tribal gov-ernments. Healthy Families is a child abuse prevention and child wellness and development program. Introduction and Background 2 OFFICE OF THE AUDITOR GENERAL voluntary. However, nationally, 90 percent of families offered Healthy Families services enroll in the program. The Healthy Families Program is based on home visitation. Home visitors regularly visit families to offer support, educa-tional materials, and referrals to various medical, mental health, and social services. (For information regarding home visitor training see the Training, Quality Assurance, and Credentialing section on page 5.) During the home visit, home visitors help families develop family support plans in which parents identify goals for themselves and their children and encourage positive parent-child interaction to strengthen bonds and promote devel-opment. Home visitors also determine which services a family might need, and provide educational material on child health, developmental milestones, safety, discipline, and nutrition. In addition to home visits, the program holds parent and play group meetings to provide participants with the opportunity to meet other families in their community. Families may participate in the program for five years. Program Stresses Early Screening and Intervention Healthy Families establishes its family visitation within the first three months after a child is born. By doing so, the program at-tempts to help families establish positive parent-child relation-ships and positive child development outcomes early on. To identify families most at risk, Healthy Families coordinates with hospitals to screen mothers giving birth within the program's service area (see Figure 1, footnote 1, page 8). To be eligible for program enrollment, participants cannot have any substantiated reports of child abuse on file with Child Protective Services (CPS) and must reside in a Healthy Families service area. The intake process is completed in two stages. Specifically, n The first stage includes a screening in which parents are iden-tified as possibly at risk for child abuse based on a combina-tion of factors such as inadequate income, unstable housing, lack of a high school diploma, inconsistent or late prenatal care, and being unmarried or separated. The screening proc-ess is usually completed while the mother is still in the hospi-tal. The program refers to those screenings that identify at- The program provides services through home visitation. Participants are enrolled within the first three months after their child is born. Introduction and Background 3 OFFICE OF THE AUDITOR GENERAL risk parents as positive screens. During fiscal year 2000, Healthy Families screened 17,005 families. Of this total, 49 percent (8,382) screened positive. n During the second stage, parents who receive a positive screen are contacted by a Healthy Families worker about completing an assessment. If the parent consents to complet-ing the assessment, he or she is interviewed about topics in-cluding family history, history of substance abuse or criminal activity, stress, self-esteem issues, expectations for the child, and plans regarding discipline. If the parent receives a posi-tive assessment, indicating a potential risk for committing child abuse, he or she is invited to enroll in the program pending a CPS check. Assessments are completed a few days after the screening process either in the hospital, in the par-ent's home, or over the telephone. Families who received a positive screen and were not offered the assessment include those who were ineligible for the pro-gram because they have a substantiated abuse report on file with CPS, could not be reached, or lived in an area in which the Healthy Families Program was full. During fiscal year 2000, of those families contacted and offered the assessment, 1,069 completed the assessment and 533 refused the assess-ment. Nine-hundred thirty-one families assessed positive for child abuse risk and 854 of these families enrolled in the pro-gram. Once enrolled in the program, Healthy Families participants are required by statute to perform community service in exchange for program services. The program has established that families must complete 12 hours of community service per year of par-ticipation. Home visitors at each site are responsible for serving 12 to 25 families. The frequency of home visits varies according to the family's level of participation. Visits are more frequent at the beginning of the family's participation and less frequent as the family becomes more self-sufficient. Increased self-sufficiency is determined by a variety of factors such as a stable home envi-ronment, ability to access resources independently, and utiliza-tion of support networks. Introduction and Background 4 OFFICE OF THE AUDITOR GENERAL Program Goals Five program goals are outlined in A.R.S. §8-701. Healthy Fami-lies goals and the procedures used by the program to address these goals are listed below: n Goal 1: Reduce child abuse and neglect—To reduce child abuse and neglect, home visitors assist families with enhanc-ing parent-child interaction, attachment, and bonding. They provide information about child safety, discipline, and meth-ods for anger management, and assess parent stress levels and home safety. As appropriate, home visitors may provide referrals for counseling and treatment services to address substance abuse and domestic violence. n Goal 2: Promote child wellness and proper develop-ment— Home visitors provide information on child health and nutrition and encourage families to have a primary care physician, to receive well-baby check-ups, and to have their children immunized. They also conduct periodic develop-mental assessments and provide referrals to medical provid-ers and the Arizona Early Intervention Program (AzEIP) for children with potential delays. n Goals 3 and 4: Strengthen family relations and promote family unity—Healthy Families uses an approach that fo-cuses services on the family and building on family strengths. They encourage positive parent-child interaction and discuss family relationship issues with participants. When appropri-ate, they refer participants to counselors to discuss family re-lationship issues. Although most primary caregivers who en-roll in the program are mothers, the program encourages fa-thers and other family members to participate in home visits and group activities. n Goal 5: Reduce dependency on drugs and alcohol—To address issues of substance abuse, Healthy Families began administering a substance abuse assessment instrument in July 1998. When alcohol or drug abuse problems are identi-fied, home visitors are trained to discuss the issue with the participant and/or provide a referral for treatment. Introduction and Background 5 OFFICE OF THE AUDITOR GENERAL Training, Quality Assurance, and Credentialing Home visitors are trained home visiting specialists. They must have at least a high school diploma to be employed by the pro-gram, but many also have college degrees. Home visitors receive a minimum of 30 hours of training a year in subjects such as child development, substance abuse, and identifying and report-ing child abuse and neglect. New home visitors are required to complete four and one-half days of initial training. New staff also complete self-guided training modules under the supervision of site managers. Initial training includes methods for identifying child abuse and neglect, infant growth and development, and methods for interacting with families and encouraging positive parent-child interaction. Finally, new staff must also complete on-the-job training in which they observe home visits conducted by more experienced staff. To help ensure that Healthy Families’ sites are in compliance with program policies and procedures, DES contracts with an evaluation firm to provide quality assurance and database man-agement. Specifically, quality assurance coordinators from this firm visit each site at least twice a year to review participant files for accuracy. This firm also collects and maintains all program data from each Healthy Families site and is responsible for test-ing this data to ensure that it is reliable and accurate. In April 2000, Healthy Families—Arizona became the first pro-gram in the nation to receive a statewide credential from Prevent Child Abuse America and the Council on Accreditation. During the credentialing process, the program had to demonstrate ad-herence to national standards by submitting information on training, technical assistance, policies, quality assurance, evalua-tion, and administration. Each site completed a self-assessment of its performance according to 12 critical elements and 138 stan-dards of best practice. Seven sites received formal visits from national credentialing committee representatives. During the site visits, committee representatives evaluated the quality of service provided by each site. The credential attests that Healthy Families—Arizona meets nationally established research-based standards for quality ser- Healthy Families—Arizona was first in the nation to re-ceive a statewide credential. Introduction and Background 6 OFFICE OF THE AUDITOR GENERAL vice delivery, and best practice standards for management and administration. Contractors and Appropriations Healthy Families provides services through various contractors, which include medical centers, local health departments, and social service agencies. In fiscal year 2000, DES had 11 contracts with 10 separate contractors. These contractors provide services at 23 sites in 10 counties (see Figure 1, page 8). For fiscal year 2000, two new sites were added in Maricopa County. In addi-tion, the Pascua Yaqui Tribe site offered services in previous years through a federal grant but was awarded program funding for the first time this year. Each site serves selected areas within a 40-mile radius of the contractor's office. These areas, identified by their zip codes, are chosen based on the number of live births per year, the number of CPS reports for children ages 0 to 5 years, and other factors, including low income and under-utilization of health care services. Additionally, Healthy Families contracts with an evaluation firm for database management, evaluation, quality assurance, and training. For fiscal year 2000, program funding for Healthy Families to-taled about $5.6 million, $4 million of which was from the State’s General Fund appropriation. In fiscal years 1997, 1998, and 1999 the General Fund appropriation totaled about $3 million for each fiscal year. Other funding sources for the program include DES' Child Abuse Prevention Fund, federal grants, contractor contri-butions, and private donations (see Table 1, page 7). Scope and Methodology A.R.S. §41-1279.08 mandates that the Office of the Auditor Gen-eral evaluate the Healthy Families Program. The statute specifies that the evaluation will include an assessment of the program's effectiveness in achieving its goals. According to the statute, additional evaluation requirements include providing informa-tion about the level and scope of program services, enrollment eligibility, participant demographic characteristics, long-term Healthy Families provides services through local contrac-tors. Introduction and Background 7 OFFICE OF THE AUDITOR GENERAL Table 1 Healthy Families Program Schedule of Distributions by Funding Source and Contractor, and Average State Contribution per Family 1 Year Ended June 30, 2000 (Unaudited) Funding Source Contractor: State General Fund Child Abuse Prevention Fund Tobacco Taxes 2 Federal Grants 3 Total Southwest Human Development $1,098,787 $179,892 $132,580 $195,837 $1,607,096 Child and Family Resources: Urban 769,980 106,530 213,923 1,090,433 Rural 539,699 184,819 183,569 908,087 Coconino County Department of Pub-lic Health 296,761 33,506 31,586 1,325 363,178 Yavapai Regional Medical Center 215,587 22,773 25,534 263,894 Pinal County Department of Public Health 171,605 45,512 24,735 241,852 Lake Havasu Social Services Inter-agency Council 190,772 20,470 20,470 231,712 Yuma County EXCEL Group 108,903 40,743 20,008 169,654 Verde Valley Medical Center 88,875 30,415 16,590 135,880 Pascua Yaqui Tribe 8,254 38,730 27,792 74,776 Parents Anonymous of Arizona (Tuba City) 32,940 20,790 10,530 64,260 LeCroy & Milligan (provides evalua-tion, quality assurance, and training statewide) 445,043 36,832 481,875 Total monies distributed $3,967,206 $ 724,180 $ 716,357 $224,954 $5,632,697 Average state contribution per family 4 $ 3,164 $ 577 $ 571 $ 4,312 1 Department-distributed monies to contractor, rather than contractor expenditures, are presented. Contractors do not report expenditures in a timely manner and funding sources appear to be approximate expenditures of Department-distributed monies. In addition, the schedule excludes the required contractor matches of at least 10 percent, because con-tractors do not report the value of their noncash resources, such as office space and personnel, on a consistent basis. 2 The program received monies from the Parent’s Commission on Drug Education and Prevention beginning in January 2000. The Commission’s funding comes from tobacco taxes; therefore, this schedule presents the source of funding as to-bacco taxes. 3 Consists of approximately $164,400 and $60,600 received from the Community Based Family Resource and Support and the Safe and Stable Families Act federal grants, respectively. 4 Calculation based on the total number of families served (1,254) during the fiscal year, including families who have disenrolled. Calculation excludes the federal grants and contractor contributions to arrive at the State’s contribution per family. Source: Auditor General staff analysis of financial information provided by the Department of Economic Security. Introduction and Background 8 OFFICE OF THE AUDITOR GENERAL Figure 1 Healthy Families Program Contract Provider Locations1 Year Ended June 30, 2000 u Coconino County Health Serving Page and Flagstaff areas u Verde Valley Medical Center Serving Verde Valley area u Yavapai Regional Medical Center Serving Prescott area u Inter-Agency Council Serving Lake Havasu and Parker areas u Southwest Human Development Serving 7 sites in Maricopa County u Excel Group Serving Yuma area u Pinal County Health Serving Casa Grande and Coolidge areas Child & Familyuu Resources Serving 4 sites in Metro Tucson u u Child & Family Resources Serving Nogales Sierra Vista, and Douglas/Bisbee areas u 1 Each site serves selected areas within a 40-mile radius of the contractor’s office. These areas, identified by their zip codes, are chosen based on the number of live births per year, the number of CPS reports for children ages 0 to 5 years, and other factors including low income and under-utilization of health care services. Source: Auditor General staff analysis of Department of Economic Security Healthy Families contracts for the year ended June 30, 2000. Pascua Yaqui Tribe Parents Anonymous Serving Tuba City u u ¿ Introduction and Background 9 OFFICE OF THE AUDITOR GENERAL savings associated with the program and rates of enrollment and disenrollment. During this annual evaluation, Auditor General staff visited 11 of 23 sites. Site visits included: n An interview with site manager(s); n A group interview with home visitors and assessment work-ers; n Accompanying home visitors on 1 to 2 home visits. During home visits, evaluators interviewed 13 program participants and conducted 13 structured observations of the visit; and n Reviews of 110 participant files to check the accuracy of Healthy Families databases. Specifically, information ob-tained from the file reviews was compared to information in the program's database. This comparison showed that data was reliable. Evaluators also analyzed data from the following assessment tools: n The Parenting Stress Index, which provides a measure of parental stress; n The Child Safety Checklist, which measures the safety of the home environment; n The Ages and Stages Questionnaire, which assesses children for potential developmental delays. A description of the assessment tools is included in the Appen-dix (see pages a-i through a-ii). In addition to the assessment tools, evaluators collected and analyzed: n Approximately 83,000 records of substantiated and unsub-stantiated CPS reports to determine the number of reports re-ceived by program participants and members of the com-parison group; Introduction and Background 10 OFFICE OF THE AUDITOR GENERAL n Immunization rates of children in the program; n Healthy Families participant usage of two types of public assistance: Temporary Assistance for Needy Families (TANF) and food stamps. In conducting the analyses of child abuse rates, evaluators com-pared substantiated and unsubstantiated report rates for Healthy Families participants with those of a comparison group. Healthy Families participants included in the analysis of abuse rates are those participants who enrolled in the program from July 1, 1997 to November 30, 1999, received at least four home visits, and participated in the program for at least six months. The compari-son group comprised individuals who enrolled in Healthy Fami-lies between July 1, 1997 and November 30, 1999, but left the program before receiving four home visits. Evaluators also ana-lyzed CPS data to determine how many participants and com-parison group members were referred to Family Builders, which is a program that provides services to families who have been reported to CPS for low and potential risk of child abuse. The evaluation includes findings and recommendations in the following areas: n Child abuse rates for Healthy Families participants and the comparison group are not significantly different; n The Healthy Families program succeeds in providing health referrals, encouraging on-time immunizations, and promo-ting safety. n Parenting stress measure shows improved family relations. In addition to these finding areas, the evaluation also contains an Other Pertinent Information section (see pages 31 through 34), regarding the implementation of new methods for measuring program outcomes. These changes address recommendations made in the last Auditor General Report (Report No. 00-1). Introduction and Background 11 OFFICE OF THE AUDITOR GENERAL Acknowledgements The Auditor General and staff express appreciation to the Direc-tor of the Department of Economic Security and DES staff in the Division of Children, Youth, and Families and the Office of Evaluation; the Manager of the Office of Prevention and Family Support, Healthy Families Statewide Coordinator, and Program Specialist; Healthy Families Data Management, Evaluation and Quality Assurance staff; Healthy Families supervisors, home visitors, and family assessment workers; and the Department of Health Services' Bureau of Epidemiology and Disease Control for their cooperation and assistance during this evaluation. 12 OFFICE OF THE AUDITOR GENERAL (This Page Intentionally Left Blank) 13 OFFICE OF THE AUDITOR GENERAL FINDING I HEALTHY FAMILIES PARTICIPANT CHILD ABUSE RATES DO NOT DIFFER SIGNIFICANTLY FROM THE COMPARISON GROUP For the second year in a row, comparison of child abuse rates for Healthy Families participants and families who left the program show that abuse rates were not significantly different. While one possible explanation of these findings might be that the program does not have a significant effect on child abuse, other explana-tions may also account for the similarity in outcomes between the two groups. Child Abuse Rates for Healthy Families Participants and Comparison Group Similar Child abuse rates for families who participated in the Healthy Families Program did not differ significantly from child abuse rates for a comparison group of families who left the program after only a short time. Additionally, substantiated reports for many participants and control group members occurred a year or more after program enrollment. Finally, abuse rates found in this report are consistent with rates found in the Auditor Gen-eral’s last evaluation (see Report No. 00-1). Evaluation compared Healthy Families participants with others who left the program—To assess the program’s effect on child abuse, evaluators reviewed CPS records in the Children’s Infor-mation Library Data Source (CHILDS).1,2 The analysis included 1 The data system contains abuse report information such as the alleged abuser, report date, reporting source, and investigation findings. 2 Auditor General staff also reviewed the CHILDS system’s intake, investi-gation, and security procedures to determine whether adequate controls for ensuring the reliability and accuracy of data were in place. Staff con-cluded that such controls were in place. Finding I 14 OFFICE OF THE AUDITOR GENERAL all substantiated and unsubstantiated CPS reports made at least six months after intake in Healthy Families for participants and comparison group members. This time period was selected to allow participants to have sufficient program exposure before assessing whether the program had an impact. For instance, for families with intake dates of July 1, 1997, evaluators examined reports made on or after January 1, 1998. Evaluators also exam-ined the proportion of participants and comparison group mem-bers who received referrals to Family Builders during the same period. The Healthy Families participant group comprised families who received at least four home visits and participated in the pro-gram for at least six months; members of the comparison group left the program before receiving four home visits. The analysis included 1,139 program participants and 512 families in the comparison group. No significant difference between Healthy Families and com-parison group—The analysis revealed that Healthy Families participants and comparison group members had similar rates of CPS reports. For substantiated reports—those in which DES concluded that abuse occurred—the percentage of families who had a report on file was 1.7 for program participants, and 1.8 percent for the comparison group. The slight difference between the two is not statistically significant. For unsubstantiated re-ports, program participants had a slightly higher rate (6.1 percent versus 5.5 percent for the comparison group), but again the dif-ference was not statistically significant. As a further point of comparison, evaluators also analyzed the percentage of program participants and comparison group members who were referred to Family Builders. Family Builders is a program that provides services to families who have been reported to CPS for low and potential abuse risk. Once referred to Family Builders, the report is closed and CPS does not conduct an investigation of the case.1 Results show that 1.8 percent of program participants and 2.7 percent of the comparison group 1 To be eligible for Family Builders, the family cannot have an existing open file with CPS. Additionally, the case cannot involve allegations of sexual abuse, or current injuries, or involve a child who is currently in out-of-home placement or a ward of the State. Participant abuse rates do not differ from comparison group. Finding I 15 OFFICE OF THE AUDITOR GENERAL were referred to Family Builders. However, this difference is also not statistically significant. Results are summarized in Table 2. Table 2 Healthy Families Program Percentage of Participants and Comparison Group Members with Substantiated and Unsubstantiated Reports of Abuse and Referrals to Family Builders Reports Received Between January 1, 1998 and May 31, 2000 Outcome Measure Program Participants Comparison Group Substantiated reports of abuse 1 1.7% 1.8% Unsubstantiated reports of abuse 1 6.1 5.5 Referred to Family Builders 1 1.8 2.7 1 Differences are not statistically significant. Source: Auditor General staff analysis of data provided by Child Protective Services and the Healthy Families Program. Many reports received one year after intake—The majority of program participants and comparison group families with sub-stantiated reports of abuse received these reports at least one year after their initial intake into the Healthy Families Program. Over half the reports were received two years after intake.1 Moreover, the results of this analysis also show that 79 percent of Healthy Families participants who have a substantiated report in the CHILDS database received the report after they left the pro-gram. Comparison with prior Auditor General reports—The rates found in this report are consistent with those found in the Audi-tor General’s last evaluation (see Report No. 00-1). However, an Auditor General evaluation issued in January 1998 found higher 1 For this analysis, evaluators included all substantiated reports that oc-curred at any point after intake—that is, they did not limit the analysis to reports that occurred after six months of exposure to the program. Finding I 16 OFFICE OF THE AUDITOR GENERAL abuse rates (see Report No. 98-1). Specifically, 3.3 percent of pro-gram participants and 8.5 percent of comparison group members received substantiated reports of abuse (see Table 3). These find-ings show a decrease in rates over the last two years. Possible Explanations of Results There are several possible explanations for the evaluation results. First, it is possible that the effects of the program on preventing child abuse cannot be adequately measured by the current evaluation design because of various factors associated with changes in abuse reporting requirements. Second, it is possible the program’s impact is lessened because most program services occur before the time period when the risk of child abuse in-creases. Third, the program may not have an effect on prevent-ing child abuse. Explanation 1: Program reduces child abuse risk but vari-ous factors might have affected the results—Compared to the 1998 evaluation, Healthy Families participants and compari-son group members were found to have lower rates of abuse in Table 3 Healthy Families Program Percentage of Participants and Comparison Group Members with Substantiated Abuse Reports Office of the Auditor General Evaluations Issue Date Healthy Families Participants Comparison Group January 1998 a 3.3% 8.5% February 2000 b 1.6 1.4 February 2001 b 1.7 1.8 a Difference between Healthy Families participants and comparison group statistically significant at the .001 level. That is, the probability that the difference in rates occurred by chance is less than 1 in 1,000. b Difference between Healthy Families participants and comparison group not statistically significant. Source: Auditor General staff analysis of data provided by Child Protective Services and the Healthy Families Program. There are several possible explanations for evaluation results. Finding I 17 OFFICE OF THE AUDITOR GENERAL the current and last evaluations. The reduction in abuse rates for both groups may reflect changes in CPS child abuse substantia-tion standards, changes in the construction of the comparison group, and changes in program enrollment that occurred after the 1998 evaluation. These changes may have affected the evaluation design, limiting the ability of evaluators to assess the program’s impact on child abuse. Specifically, n Overall there is a lower occurrence of substantiated abuse, due in part to the appeals process—Effective January 1, 1998, DES changed its procedures for substantiat-ing reports of abuse and neglect. Under A.R.S. §8-811, the Department created the Protective Service Review Team (PSRT). The PSRT reviews reports of abuse and/or neglect to determine if report allegations should be substantiated in the Department’s Central Registry.1 Since the implementation of the PSRT, substantiation rates have decreased. According to a DES report, 25 percent of CPS cases were substantiated in fis-cal year 1999, the first full fiscal year with the appeals process in place.2 In comparison, 45 percent of cases were substanti-ated in fiscal year 1997, the last fiscal year without the ap-peals process. n Comparison group different—The current comparison group comprised families who left the program before receiving four home visits. This comparison group differs from the group used in the 1998 evaluation, which was con-structed using individuals eligible for Healthy Families but who could not enroll because the program was full. Al-though current comparison group families did not receive four home visits, many received one to three visits. During these initial visits, Healthy Families staff explain the pro-gram’s mandatory CPS reporting policy and provide infor-mation about child safety (for example, the importance of never shaking a baby, or leaving a baby unattended). Within 1 PSRT reviews do not include those cases in which criminal or civil charges have been filed against the alleged abuser. 2 Department of Economic Security. Child Welfare Reporting Requirement: Annual Reports for the Period July 1, 1996 through June 30, 1997, and the Period of July 1, 1998 through June 30, 1999. Finding I 18 OFFICE OF THE AUDITOR GENERAL the first three visits these families can also receive various re-ferrals for family assistance. In addition, in the current analysis, there are some small but statistically significant differences between participants and the comparison group. Participants are slightly older, less likely to be single, and less likely to be employed. Participant scores on the Family Stress Checklist Risk Assessment are also higher by an average of one point. No significant differ-ences were found when comparing annual household in-come, education level, or ethnicity. n Families with abuse history excluded from the pro-gram— Effective June 1, 1998, the Healthy Families statute (A.R.S. §8-701) was changed, prohibiting the enrollment of families with prior substantiated CPS abuse reports. Families who received a substantiated report, but were already en-rolled in the program, were also required to exit.1 Thus, fami-lies with a demonstrated propensity for abuse are no longer enrolled in the program. Explanation 2: Program impact on abuse may be affected by timing and length of participation—Evaluators’ preliminary analysis of report dates for substantiated abuse reveals that the majority of program participants and comparison group mem-bers who receive reports do so one year or more after the fam-ily’s initial exposure to the program. In most cases, the abuse reports for program participants also occurred after the family left the program as a majority of families participate for 12 or fewer months. Therefore, it is possible that the risk of child abuse may increase as the child gets older, but that participants have already left the program before this increase occurs. 1 Comparison of initial assessment scores for parents who entered the pro-gram before June 1, 1998, with scores of those parents who entered after June 1, 1998, suggest current enrollees are less at-risk. Specifically, Family Stress Checklist (FSC) scores are lower for participants who entered the program after the enrollment criteria changed. The average FSC score is 38.6 for pre-June 1, 1998, enrollees and 36.4 for post-June 1, 1998, enrollees. This difference is significant at the .001 level. Scores of 25 or higher indicate a potential risk to commit abuse. Finding I 19 OFFICE OF THE AUDITOR GENERAL Explanation 3: Groups do not differ because program might not reduce child abuse risk—Because last year’s and this year’s evaluations show no difference in program participant and comparison group abuse rates, it could be concluded that the program does not reduce child abuse risk. However, before making this determination, evaluators would prefer to ad-dress the other possible explanations. This would involve constructing additional comparison groups of “at-risk” fami-lies who have not received any child abuse prevention ser-vices. Additionally, because abuse often occurs a year or more after program enrollment, other measures of the pro-gram's long-term effect will have to be developed. However, establishing new comparison groups and developing addi-tional long-term measures would require extensive efforts and the results of the further analyses would likely not be available for several years. 20 OFFICE OF THE AUDITOR GENERAL (This Page Intentionally Left Blank) 21 OFFICE OF THE AUDITOR GENERAL FINDING II HEALTHY FAMILIES PROGRAM PROVIDES HEALTH REFERRALS, ENCOURAGES ON-TIME IMMUNIZATIONS, AND PROMOTES SAFETY Although the Healthy Families Program’s long-term effect on preventing child abuse cannot be clearly determined, it has been successful in providing health referrals, encouraging on-time immunizations, and promoting safety. More specifically, on average, the majority of children in the program are fully immu-nized and most children have a primary health care provider. Additionally, most children receive regular developmental as-sessments. Parents also improved their compliance with home safety measures over time. Background According to program goals defined in the Healthy Families statute (A.R.S. §8-701), the program is required to provide par-ents with information about child development and preventative health care. Program goals also include reducing participant substance abuse. To determine if the program is achieving these goals, evaluators examined several health, safety, and develop-ment measures. The measurement tools used to collect this in-formation are described in the Appendix (see pages a-i through a-ii). Although program goals include reducing participant substance abuse, evaluators reported in the February 2000 evaluation (see Auditor General Report No. 00-1) that the program’s impact on reducing participants’ dependency on drugs and alcohol could not be determined. Specifically, evaluators reported in the Finding II 22 OFFICE OF THE AUDITOR GENERAL February 2000 evaluation that the CAGE1 substance abuse screen used by the program was not an adequate measure for assessing change in drug and alcohol usage and recommended replacing this instrument. To address this recommendation, the Healthy Families Program began using a new substance abuse question-naire in July 2000 (see Other Pertinent Information, pages 31 through 34). However, data from the questionnaire will not be available for analysis until the 2001 evaluation. Participants Receive Developmental Screenings, Referrals, and On-Time Immunizations Evaluators found that consistent with the January 1998 and Feb-ruary 2000 evaluations, the majority of children in the Healthy Families Program received age-appropriate immunizations and developmental screenings. Most participants also had a primary health care provider. Results of the developmental assessments show that most children in the program are developing nor-mally. However, for those children identified as potentially hav-ing delays, the program provides referrals to appropriate ser-vices and recently began tracking those referrals in its new fam-ily update form (see Other Pertinent Information on page 31). Healthy Families monitors immunization rates and promotes preventative medical care—Eighty-five percent of two- to three-month- old children enrolled in the program are fully immu-nized. At four months of age, 73 percent of children in the pro-gram are fully immunized; at 6 months of age, 66 percent are fully immunized; and at 1 year to 15 months of age, 73 percent are fully immunized. Generally, these rates are higher than esti-mated vaccination rates for children served by local public health clinics in the same areas as program sites. Additionally, these 1 The acronym “CAGE” is derived from the first letter in each of four ques-tions that are asked of the respondent: (1) Have you every felt the need to cut down on drinking/drug use? (2) Have you ever felt annoyed by oth-ers’ criticism of your drinking/drug use? (3) Have you ever felt guilty about your drinking/drug use? (4) Have you ever had a drink/taken drugs first thing in the morning (eye-opener)? Two or more “yes” re-sponses are considered indicative of a substance abuse problem. Finding II 23 OFFICE OF THE AUDITOR GENERAL rates are comparable to statewide rates for Arizona reported by the Centers for Disease Control and Prevention. In addition to encouraging parents to ensure that their children are fully immunized, the program also emphasizes the impor-tance of preventative and routine medical care. At 2 months in the program, 97 percent of children had a medical primary care provider; and at 6 and 12 months, 98 percent of children had a medical provider. To promote positive health outcomes, home visitors ask about immunization schedules for the children they serve, provide educational information about immunizations, and refer families to health/immunization clinics. In addition, program staff em-phasize the importance of routine and preventative medical care and encourage participants to use referrals to obtain a primary care provider. Healthy Families program conducts developmental assess-ments— Home visitors use the Ages and Stages Questionnaire (ASQ) to conduct regular assessments of the physical and social development of children enrolled in the program. The ASQ is administered at the following ages: 4, 6, 12, 18, 24, 30, 36, and 48 months. In addition to conducting these assessments, home visi-tors provide families with information about the developmental stages to alert parents to signs of potential delays and reduce fears associated with unrealistic expectations. Program partici-pants cited developmental assessments as important for identify-ing milestones, early detection of potential delays, and reducing confusion about their child’s development. Results of the ASQ reveal that most children in the program are developing at levels that are appropriate for their age (see Table 4, page 24). The ASQ assesses five developmental areas— communication, gross motor skill, fine motor skill, problem solv-ing, and personal social development. If a child receives a score below the minimum level for normal development, home visi-tors are trained to explain the result to the parent and provide referrals to a primary health care provider, or the Arizona Early Intervention Program (AzEIP), to arrange for further assessment. However, as with the previous Auditor General report (see Re-port No. 00-1), evaluators were unable to directly link develop- Program conducts develop-mental assessments of partici-pants’ children. Finding II 24 OFFICE OF THE AUDITOR GENERAL mental screenings with referrals to AzEIP or a health care pro-vider. The program has revised its family update form. The up-date form now contains questions which enable the program to document if children with developmental delays are offered the appropriate referrals. This change is discussed in the Other Per-tinent Information section of this report (see pages 31 through 34). Table 4 Healthy Families Program Ages and Stages Questionnaire Percentage of Children Developing at Age-Appropriate Levels January 1994 to May 2000 Age in Months Type of Development 4 6 12 18 24 30 36 48 Communication 98.4% 99.8% 98.6% 89.8% 89.4% 91.8% 92.4% 96.7% Gross motor skills 91.7 98.5 97.5 99.2 95.6 95.9 98.2 98.9 Fine motor skills 96.5 99.0 98.8 98.9 96.0 91.2 95.5 98.9 Problem solving 96.7 99.1 96.0 98.6 93.8 93.9 89.2 81.9 Personal/Social 96.5 98.6 97.0 99.9 93.3 94.7 94.6 100.0 Number of children1 1,884 1,849 1,183 741 499 342 224 94 1 The number of children varies by scale; the number reported is the largest number assessed for that age group. Source: Auditor General staff analysis of data provided by the Healthy Families Program. Participants Show Significant Improvements in Home Safety The safety of program participants’ homes improves over the course of their enrollment in the program. Healthy Families staff administer the Child Safety Checklist (CSC) to identify potential safety hazards and also discuss with parents ways to improve home safety. On average, program participants improved their home safety after one year by taking steps to comply with items on the CSC. Finding II 25 OFFICE OF THE AUDITOR GENERAL Healthy Families participants maintain homes that meet child safety requirements—Evaluators analyzed results from the CSC at 2, 6, and 12 months and found that home safety improves after a year of enrollment. On average, the percentage of child safety measures implemented improves from 87 percent to 94 percent between 2 to 12 months (see Table 5). Analyses of individual items on the checklist show significant improvement in key safety measures such as covering outlets and keeping poisons and small items that are potential choking hazards out of reach (see Appendix, page a-ii, for a description of the checklist). Nearly all of the program participants indicated initial (2 months) and subsequent (6 and 12 months) compliance with other important safety measures such as using car seats, making sure their child is never alone in the house or car, and keeping dangerous objects such as scissors and matches out of their child’s reach. Table 5 Healthy Families Program Percentage of Safety Measures Implemented by Age of Child July 1998 to May 2000 Age of Child Percentage Implemented 1 2 months 87% 6 months 91 12 months 94 1 Increases in percentages of safety measures implemented are all statisti-cally significant at the .001 level. That is, the probability that the average gain in compliance occurred by chance is less than 1 in 1,000. Source: Auditor General staff analysis of data provided by the Healthy Fami-lies Program. Participants improve home safety. Finding II 26 OFFICE OF THE AUDITOR GENERAL Recommendations 1. To ensure compliance with its statutorily defined goal to reduce substance abuse, the Healthy Families Program should continue to implement the new questionnaire for as-sessing participant substance abuse problems and method for tracking substance abuse referrals. 2. To ensure that families with children who receive scores in the delayed range on the ASQ receive appropriate referrals, the Healthy Families Program should continue to implement new measures for tracking these referrals. 27 OFFICE OF THE AUDITOR GENERAL FINDING III PARENTING STRESS MEASURE SHOWS IMPROVED FAMILY RELATIONS Results from the Parenting Stress Index show that parental stress decreases after 12 months of participation. Evaluators also found that parents with higher scores on the restricted role, isolation, and depression subscales were more likely to receive referrals to mental health services compared with those whose scores indi-cated lower risk. To assist families in improving parent-child interaction, home visitors provide emotional support and infor-mation to help parents understand their child’s behavior. Background To address recommendations in the last Auditor General evalua-tion, the Healthy Families program discontinued using the Home Observation for Measurement of Environment (HOME) to assess family functioning (see “Other Pertinent Information,” pages 31 through 34). In place of the HOME, the program is using the Parenting Stress Index (PSI) to measure family rela-tions. The PSI is first administered to program participants after 3 weeks of enrollment and then again at 6 and 12 months. Cur-rently, there is no comparison group for this measure. Participants’ Stress Reduced After 12 Months Although scores for most Healthy Families participants are within normal ranges, after 12 months in the program partici-pants show statistically significant decreases in parental stress on 6 of the PSI’s subscales. As shown in Table 6, stress related to attachment, role restriction, competence, social isolation, depres-sion, and mood is lower at 12 months than it is at 3 weeks. Parents have reduced stress after 12 months in the pro-gram. Finding III 28 OFFICE OF THE AUDITOR GENERAL Research has shown that high scores on parenting stress are related to the potential to abuse one’s child. Conversely, low to normal scores are related to positive parent-child relationships. Thus, reductions in parental stress suggest that program services may improve family relations. Table 6 Healthy Families Program Average Parenting Stress Scores After 3 Weeks and 12 Months in the Program July 1998 to May 2000 Participant Scores Scale1 Normal Stress Range 3 Weeks 12 Months Competence2 23-34 31.9 29.9 Attachment2 10-15 13.1 12.0 Role Restriction2 14-23 20.0 18.5 Depression 2 16-24 20.2 19.1 Isolation 2 10-16 14.1 13.2 Distractibility 3 20-28 25.3 25.0 Mood 2 7-11 10.9 9.4 1 Higher numbers indicate higher stress levels. See Other Pertinent Information (pages 31 through 34) for detailed information on these scales. 2 Differences from 3 weeks to 12 months are statistically significant at the .05 level or bet-ter. That is, the chance that the average reduction in parenting stress occurred by chance is less than 5 in 100. Evaluators also analyzed participant PSI scores at 6 months. For the competence, role restriction, and mood subscales evaluators found significant differences between scores at 3 weeks and 6 months. Scores for the depression and attachment sub-scales at 6 months were significantly different from scores at 12 months. 3 Differences not statistically significant. Source: Auditor General staff analysis of data provided by the Healthy Families Program. Finding III 29 OFFICE OF THE AUDITOR GENERAL Parents With Higher Stress More Likely to Receive Mental Health Referrals Additionally, participants who scored in the higher risk ranges for depression, isolation, and role restriction were more likely than families scoring at lower risk levels to receive referrals for mental health services. This shows that the PSI may also be a useful tool for identifying families who need additional services. Home Visitors Provide Information on Parent-Child Bonding To assist families with parent-child interaction, home visitors provide emotional support and information to help parents un-derstand their child’s behavior. Information provided can in-clude methods for alleviating stress, activities to improve parent-child interaction and bonding, and referrals to counseling ser-vices to assist with family relationship problems. 30 OFFICE OF THE AUDITOR GENERAL (This Page Intentionally Left Blank) 31 OFFICE OF THE AUDITOR GENERAL OTHER PERTINENT INFORMATION The previous report issued by the Office of the Auditor General (see Report No. 00-1) offered several recommendations to im-prove the measurement of the Healthy Families Program’s effect on family functioning and reduction of substance abuse. It is necessary for the program to continue collecting data on the program’s effect on family relations and reducing dependency on drugs and alcohol because these are defined in statute (A.R.S. §8-701) as two of the program’s five goals. The previous evalua-tion report also recommends tracking referrals for children with developmental delays. To address these recommendations, the program has implemented several changes. Healthy Families stopped using the HOME—During last year’s evaluation, evaluators were not able to make definitive conclu-sions about the program’s impact on family functioning because the analysis was limited by reliability problems with the Home Observation for Measurement of the Environment (HOME) assessment tool. Specifically, statistical analyses of variety, accep-tance, and organization—three of six areas assessed by the HOME—revealed that questions within each area were not strongly related to each other. This suggests that some items on the HOME do not measure the concepts they were intended to assess. Following the Auditor General’s recommendation, the program stopped using the HOME. However, because improving family relations and family unity are defined in statute as program goals, it is necessary for the program to collect information re-lated to these goals. Therefore, in place of the HOME, the pro-gram will use the Parenting Stress Index (PSI) to assess the pro-gram’s impact on family relations. Past research has shown that low scores on the PSI are related to positive parent-child relation-ships. In particular, the PSI, designed to measure “stressful par-ent- child systems,” can identify areas in which improvements are needed in parent-child relations. Below are descriptions of what high shores on each subscale indicate, according to Program stopped using the HOME assessment. Other Pertinent Information 32 OFFICE OF THE AUDITOR GENERAL Richard Abidin, PSI’s creator1. The first five scales measuring competence, isolation, attachment, restricted role, and depression focus on parent characteristics. The last two scales, measure child distractibility and child mood, focus on child characteristics. n Competence: High scores on the competence scale can be the result of the lack of child development knowledge, lim-ited child management skills, or a feeling that parenting is less reinforcing than expected. High scores may also be re-lated to “lack of acceptance and criticism from the child’s other parent.” n Isolation: Parents who score high on the social isolation scale tend to be “isolated from their peers, relatives, and other emotional support systems.” Additionally, “in many in-stances, their relationships with their spouses are distant and lacking in support for their efforts as parents.” n Attachment: High scores on attachment may indicate that parents do not feel closeness with their child or feel unable to understand the child’s feelings and needs. n Restricted Role: High scores indicate that the parent sees the child as restricting their freedom and may feel controlled by their child’s demands or needs. As a result, resentment and anger may build toward one’s child and/or spouse. n Depression: High scores on the depression scale can be indicative of clinical depression. Depressed parents may lack the energy needed to fulfill parental responsibilities. n Child Distractibility: High scores on this subscale indicate that either the child has Attention Deficit Disorder with hy-peractivity or that the child is normal, but that the parent lacks the energy to keep up with the child. n Child Mood: High scores on child mood are associated with children who are unhappy and depressed. When scores are extremely high, there may be impairment in parental at- 1 Richard R. Abidin. Parenting Stress Index: Professional Manual. Odessa, FL: Psychological Assessment Resources, Inc., 1995. Other Pertinent Information 33 OFFICE OF THE AUDITOR GENERAL tachment or parents may be absent or unavailable due to drug abuse or alcoholism. Healthy Families replaced the CAGE—Evaluators noted in the previous report that, lacking a better instrument, Healthy Fami-lies has used the CAGE substance abuse screening measure since July 1, 1998. However, the CAGE, which contains questions for identifying individuals with an alcohol and/or drug abuse prob-lem, does not allow the program to track whether a participant’s problem with substance abuse changes over time. To address the Auditor General recommendation to replace the CAGE, the pro-gram began using a new questionnaire on July 1, 2000. The new questionnaire measures the program’s effect on reducing de-pendency on alcohol and drugs over time. In addition, the pro-gram has also added questions to its family update form, which will enable staff to track referrals for substance abuse treatment. The new substance abuse assessment was adapted from the Health Screening Survey (Fleming and Barry, 1991) and includes items that measure current substance usage. For instance, par-ticipants are asked to indicate, over the past three months, the number of days per week they drank alcohol, and the number of drinks they typically have had during a single occasion. Unlike the CAGE, which only captures lifetime usage, the new assess-ment allows evaluators to assess changes in drinking patterns over time. The program will track referrals for children with delays—The previous Auditor General report noted that when children re-ceive scores in the delayed range on the Ages and Stages Ques-tionnaire (ASQ) home visitors are trained to provide a referral to a medical provider or the Arizona Early Intervention Program (AzEIP). However, because most families in the program receive medical or social services, evaluators were unable to determine the extent to which families receive services directly related to a developmental delay identified by the ASQ. To address the pre-vious report recommendation that the program track referrals for children identified as having a potential developmental de-lay, Healthy Families has revised its family update form. The form now contains questions, which supplement the ASQ, Program replaced substance abuse questionnaire. Program will track referrals for children with developmen-tal delays. Other Pertinent Information 34 OFFICE OF THE AUDITOR GENERAL and will be used to track whether children with developmental delays are offered appropriate referrals. The new form was im-plemented on July 1, 2000, and data from this measurement will be analyzed in the next evaluation. 35 OFFICE OF THE AUDITOR GENERAL STATUTORY ANNUAL EVALUATION COMPONENTS Pursuant to A.R.S. §41-1279.08, the Office of the Auditor General is required to include the following information in the annual program evaluation. C.1. Information on the number and characteristics of program participants. n Number of Participants—Since 1994, 4,421 families have participated in Healthy Families. Of these, 3,267 met program criteria for being officially engaged in the program. That is, they received at least four home visits from a home visitor. The remaining 1,154 re-ceived three or fewer home visits and were included as a comparison group when examining child abuse rates. n Mother’s Age, Marital Status, Income, Ethnicity, Education, Employment Status, and Risk Score— Information about the age, marital status, income, ethnicity, employment status, and educational status of participants is shown in Table 7 (see page 36). Be-cause in nearly all cases, the child’s mother is the pri-mary program participant, information is provided for mothers only and is presented separately for par-ticipant families and for comparison group families. Statistical analyses showed that these two groups were similar in most areas, with a few exceptions. Comparison group members were more likely than participants to be single and employed. On average, participants were also slightly older (average age of 22.9) than the comparison group (average age of 22.3). In addition, scores on the Family Stress Checklist risk assessment tool were slightly higher for participant families (average score of 38) than for comparison group families (average score of 37), and the differ-ence was statistically significant. No significant differ-ences were found when comparing annual household income, education level, or ethnicity. Statutory Annual Evaluation Components 36 OFFICE OF THE AUDITOR GENERAL Table 7 Healthy Families Program Participant and Comparison Group Member Profile January 1994 through May 2000 Program Participants N (total number of participants) =3,267 Comparison Group N (total number of comparison group members)=1,154 Mother’s age 11-15 6.0% 5.7% 16-19 34.6 37.8 20-25 34.2 36.1 26-30 13.1 11.4 31 and older 12.1 9.1 Marital status Single 68.2 76.7 Married 20.3 13.4 Divorced 3.1 3.3 Separated 3.5 3.6 Widowed .2 .3 Other/Unknown 4.6 2.8 Annual income $10,000 and under 61.9 62.5 $10,001-$15,000 20.4 19.2 $15,001-$20,000 9.0 8.7 $20,001-$30,000 5.8 7.9 $30,001 and higher 2.9 1.7 Mother’s ethnicity Hispanic 54.3 54.8 Caucasian 23.8 24.0 African-American 7.9 8.7 Native American 10.0 8.4 Asian-American .5 .2 Other 3.5 4.0 Percentage employed at intake 13.2 18.8 Full-time 6 2 . 9 6 9 . 1 Part-time 37.1 30.9 Average years of education 10.3 10.4 Average risk score on Family Stress Checklist (Range 5 to 85) 37.9 37.0 Source: Auditor General staff analysis of data provided by the Healthy Families Program. n Regions Served—The Healthy Families Program serves families in 10 of Arizona’s 15 counties. The number of participants served from each county is shown in Figure 2 (see page 37). Statutory Annual Evaluation Components 37 OFFICE OF THE AUDITOR GENERAL Figure 2 Healthy Families Program Number of Participants and Comparison Group Members by County January 1994 through May 2000 1,013 371 753 243 444 132 284 96 235 107 213 85 147 51 126 21 52 48 0 200 400 600 800 1,000 1,200 Number of Families Participants Comparison Group Members Source: Auditor General staff analysis of data provided by the Healthy Families Program. Maricopa Pima Cochise Coconino Yavapai Santa Cruz Yuma La Paz/ Pinal Mohave County n Number of Children, Household Size, Sex of Child, Health Insurance, and Public Assistance Usage—Table 8 (see page 38) presents information on program participants’ number of children, household size, and the proportion of female and male children served by the program. It also provides information on participants’ self-reported health insurance status and use of public assistance. Statistical analyses re-vealed that, on average, comparison group members had a slightly larger household size (average of 4.89) than participants (average of 4.65). No significant dif-ferences were found for the number of children in participant versus comparison group families or for the proportion of male versus female children served in each group. In addition, no group differences were found for health insurance status or for the proportion Statutory Annual Evaluation Components 38 OFFICE OF THE AUDITOR GENERAL of program participants using Temporary Assis-tance for Needy Families (TANF), food stamps, or Women, Infants and Children (WIC). Table 8 Healthy Families Program Participants and Comparison Group Members Profile of Family Characteristics January 1994 through May 2000 Program Participants Comparison Group N (total number of partici-pants) = 3,267 N (total number of comparison group members) = 1,154 Number of children 1 54.0% 55.2% 2-3 34.3 35.6 4-5 9.5 7.3 6 or more 2.1 1.8 Household size 2-3 33.0 30.6 4-5 39.2 37.4 6-7 19.4 21.1 8-9 6.5 7.2 10 or more 1.8 3.6 Sex of child Female 49.6 50.2 Male 50.4 49.8 Health insurance AHCCCS1 80.3 80.1 Private 9.3 9.1 Other 3.3 2.9 None 6.1 6.1 Unknown 1.0 1.2 Public assistance services used as reported at intake TANF2 24.7 23.3 Food Stamps 35.6 35.4 WIC3 80.1 78.9 1 Arizona Health Care Cost Containment System. 2 Temporary Assistance for Needy Families. 3 Women, Infants and Children Nutrition Program. Source: Auditor General staff analysis of data provided by the Healthy Families Program. Statutory Annual Evaluation Components 39 OFFICE OF THE AUDITOR GENERAL C.2. Information on contractors and program service providers. See Introduction and Background (page 6) for informa-tion on contractors and program service providers. C.3. Information on program revenues and expenditures. See Table 1 (page 7) for information on program revenues and expenditures. C.4. Information on the number and characteristics of enrollment and disenrollment and information from program participants on the reasons for each. As of May 31, 2000, there were 1,178 families enrolled in Healthy Families. Since 1994, 4,421 families have enrolled. Table 9 (see page 40) provides information about the length of participation for families who have exited the program. These families participated for a median of 195 days (or 6 and a-half months). Among families who were active participants as of May 31, 2000, the median dura-tion of participation was slightly more than 1 year (379 days). In addition, the January 1998 Healthy Families evaluation reported an attrition rate of 45 percent for fiscal year 1997. An attrition rate for a given fiscal year is computed by di-viding the number of families who exited the program during that year by the number who participated in the program at any point during the year. The corresponding attrition rates for fiscal years 1998, 1999, and 2000 were 42 percent, 40 percent, and 35 percent, showing a gradual improvement in retention. 1 Figure 3 (see page 40) shows the most common reasons participants exit the program. More than half terminate because they have moved or because the program could not contact the family after several attempts. Other rea-sons for termination include participants’ refusal of fur- 1 Thirty-five percent attrition rate for 2000 includes 11 months of data from fiscal year 2000 (July 1, 1999 through May 31, 2000). Statutory Annual Evaluation Components 40 OFFICE OF THE AUDITOR GENERAL ther services, refusal of a change in home visitor, and achievement of self-sufficiency. Table 9 Healthy Families Program Months of Participation for Families at Exit January 1994 through May 2000 Number Percentage 1-6 months 1,520 46.9% 7-12 months 773 23.9 13-18 months 382 11.8 19-24 months 197 6.1 25-30 months 128 3.9 31-36 months 90 2.8 37-48 months 97 3.0 49-60 months 53 1.6 Total 3,240 Source: Auditor General staff analysis of data provided by the Healthy Families Program. Figure 3 Healthy Families Program Reasons for Exit January 1994 through May 2000 n (total number of families who left the program) = 3,212 Source: Auditor General staff analysis of data provided by Healthy Families staff. Refused home visitor change 4% Self-sufficiency 9% Unable to contact 31% Refused services 14% Family moved 28% Other 14% Statutory Annual Evaluation Components 41 OFFICE OF THE AUDITOR GENERAL C.5. Information on the average cost for each participant in the program. See Table 1 (page 7) for information on the average cost for each program participant. C.6. Information concerning the progress of program participants in achieving goals and objectives. See Finding I (pages 13 through 19) for information on the program’s progress in reducing rates of child abuse and neglect. See Finding II (pages 21 through 26) for information re-garding the program’s progress in improving children’s health and participants’ home environment. Finding II includes information about immunization rates, devel-opmental screenings, child safety, and parent-child inter-action. See Section D (pages 42 through 43) for information on participants’ progress in increasing self-sufficiency through employment, and reducing their dependence on public assistance. C.7. Information on any long-term savings associated with program services. As discussed in Finding I (see pages 13 through 19), evaluators did not find statistically significant differences in the rates of substantiated child abuse reports for Healthy Families participants and the comparison group. Thus, evaluators cannot estimate potential savings asso-ciated with reduction in child abuse rates, examining such factors as costs of CPS investigations, foster care placements, and in-home services. C.8. Recommendations regarding program administra-tion. 1. To ensure compliance with its statutorily defined goal to reduce substance abuse, the Healthy Families Pro-gram should continue to implement new measures Statutory Annual Evaluation Components 42 OFFICE OF THE AUDITOR GENERAL for assessing participant substance abuse problems and tracking substance abuse referrals. 2. To ensure that families with children who receive scores in the delayed range on the ASQ receive ap-propriate referrals, the Healthy Families Program should continue to implement new measures for tracking these referrals. C.9. Recommendations regarding informational materials distributed through the program. Current and past evaluations of Healthy Families infor-mational materials show that materials distributed by the program are adequate to address program needs. Specifi-cally, Healthy Families provides program participants with information regarding medical care, mental health, employment, and education. Additionally, the program provides information to promote parent-child interaction and to assist parents with understanding their child’s de-velopment. D. Effect of the program on encouraging parental re-sponsibility in employment, self-sufficiency, and child safety. Document the income level and family size of those receiving program services. n Employment rates—As seen in Figure 4 (see page 44), employment rates for both mothers and fathers enrolled in Healthy Families generally increased over time. Healthy Families can influence these rates by providing referrals to job services. According to pro-gram data, 6.3 percent of mothers and 10.5 percent of fathers received a referral to job services within two months of enrollment, 10.2 percent of mothers and 12.8 percent of fathers received such a referral at 6 months, and 12.2 percent of mothers and 12.7 percent of fathers received a job-related referral at 12 months. Although employment rates show an increase, it is important to note that due to attrition, at each time period, there are fewer participants represented. Therefore, it is difficult to determine the extent to Statutory Annual Evaluation Components 43 OFFICE OF THE AUDITOR GENERAL which increases in employment rates are due to pro-gram participation or due to differences among peo-ple who remain in the program compared to those who drop out. n Public assistance—In order to conduct an analysis of public assistance usage, evaluators requested, data from DES,containing all records of Temporary Assis-tance to Needy Families (TANF) and food stamps re-ceived by Healthy Families participants and compari-son group members. However, after receiving the data, evaluators learned that information about sev-eral of the participants was missing. Therefore, it was determined that any analysis of public assistance us-age would provide inaccurate results. In addition, it was not possible to make additional data requests due to the substantial amount of time it would have taken to make the request, receive the data, test it for reli-ability, and analyze the results. n Child safety—See Finding II (pages 21 through 26) for information on child safety, including the results of the Child Safety Checklist assessment. n Income level and family size of those receiving program services—See C.1. (pages 35 through 38) of this section for income levels and family size of those receiving program services. Statutory Annual Evaluation Components 44 OFFICE OF THE AUDITOR GENERAL Figure 4 Healthy Families Program Employment Rates of Mothers and Fathers in Program January 1994 through May 2000 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Months in Healthy Families Program Mothers Fathers Source: Auditor General staff analysis of data provided by the Healthy Families Program. Intake 2 6 12 18 24 30 36 42 48 54 60 Percentage Employed OFFICE OF THE AUDITOR GENERAL Appendix OFFICE OF THE AUDITOR GENERAL (This Page Intentionally Left Blank) a-i OFFICE OF THE AUDITOR GENERAL Appendix Assessment Tools n Ages and Stages Questionnaire (ASQ)—The ASQ is a developmental screening tool that is completed by the parent and is used to assess whether children are developing nor-mally, both physically and socially. The questionnaire ad-dresses five areas of child development: (1) Communication, (2) Gross Motor Skills, (3) Fine Motor Skills, (4) Problem Solv-ing, and (5) Personal Social Skills. For each area, parents are asked to respond to six questions about whether their chil-dren are engaging in behavior appropriate for their age. The ASQ is administered at the following ages: 4, 6, 12, 18, 24, 30, 36, and 48 months. Reference: Bricker, Diane, Jane Squires, Linda Mounts, La- Wanda Potter, Bob Nickel, and Jane Farrell. The Ages and Stages Questionnaire: A Parent-Completed, Child-Monitoring Sys-tem. Baltimore: Paul H. Brookes Publishing Co., 1995. n The CAGE Questionnaire—Substance Abuse Screen-ing— The CAGE Questionnaire was designed to identify po-tential alcohol abuse problems. It was also modified to in-clude the abuse of drugs other than alcohol. The acronym “CAGE” stands for the first letter in each of four questions that are asked of respondents: (1) Have you ever felt the need to cut down on drinking/drug use? (2) Have you ever felt annoyed by others’ criticism of your drinking/drug use? (3) Have you ever felt guilty about your drinking/drug use? and (4) Have you ever had a drink/taken drugs first thing in the morning (eye-opener)? Two or more “yes” responses are considered indicative of a substance abuse problem. The CAGE is administered after 6 and 12 months of program par-ticipation. Appendix a-ii OFFICE OF THE AUDITOR GENERAL Reference: Mayfield, D., G. McCleod, and P. Hall. The CAGE Questionnaire: Validation of a New Alcoholism Screening Questionnaire. American Journal of Psychiatry, 1974, 131, 1121- 1123. n The Child Safety Checklist (CSC)—The Child Safety Checklist is an instrument that assesses whether various safety measures in the home have been implemented. The CSC is administered by home visitors who ask parents whether or not each safety measure on the checklist has been taken (for example, “do you use a car seat for your baby?”). There are two versions of the child safety checklist. The first is administered when the child is 2, 6, 12, 18, 24, and 30 months of age. The second contains questions designed for families with older children and is administered at the fol-lowing ages: 36, 42, 48, 54, and 60 months. n Parenting Stress Index (PSI)—The PSI is an instrument designed to identify stressful situations that could potentially put parents at risk for “dysfunctional parental behavior,” in-cluding abuse. It includes several subscales that measure stress related to child characteristics and parent functioning. For the evaluation, seven subscales were used. These in-cluded two that focused on child characteristics (child’s mood and distractibility/hyperactivity) and five that fo-cused on adult characteristics (depression, attachment, re-striction of role, sense of competence, and social isolation). The PSI is administered after 3 weeks, than at 6 and 12 months of program participation. Reference: Abidin, Richard R. Parenting Stress Index Admini-stration Manual, Third Edition. Odessa, FL: Psychological As-sessment Resources, Inc., 1995. OFFICE OF THE AUDITOR GENERAL Agency Response OFFICE OF THE AUDITOR GENERAL (This Page Intentionally Left Blank) ARIZONA DEPARTMENT OF ECONOMIC SECURITY 1717 W. Jefferson - P.O. Box 6123 - Phoenix, AZ 85005 Jane Dee Hull John L. Clayton Governor Director Ms. Debbie Davenport, CPA Auditor General 2910 North 44th Street, Suite 410 Phoenix, AZ 85018 Dear Ms. Davenport: The Department wishes to thank the Office of the Auditor General for the opportunity to respond to the recently completed audit of the Healthy Families Arizona Program. Report findings II and III indicate that the program benefits the families we serve. Finding I is inconclusive and is based on an indicator (child abuse report rates) that is a relatively low occurring event in the participant families, the comparison group, and in the general population. It is important that we keep in mind that the reason the program exists and the reason we focus resources on prevention is to help ensure that no child is maltreated and that all children have the chance to be nurtured and comforted during their growing years into adulthood. Because of the years of positive evaluation outcomes, we believe the Healthy Families Arizona Program is an effective prevention program. The Department welcomes the opportunity to work with you in helping to find other methods of determining the impact of the program’s positive outcomes on the prevention of child abuse and neglect. The Department will continue to utilize the new forms listed in the two recommendations under Finding II. Finally, please accept our appreciation for the time and effort invested in this important evaluation. We wish to specifically recognize Laurie Cohen and JoAnne Dukeshire for their dedication during the evaluation process. Sincerely, John L. Clayton DEPARTMENT OF ECONOMIC SECURITY RESPONSE TO THE HEALTHY FAMILIES ARIZONA PROGRAM EVALUATION FINDING I: Healthy Families Participant Child Abuse Rates Do Not Differ Significantly from the Comparison Group This report has found no significant difference between Healthy Families participants and the comparison group on the rates of substantiated CPS reports. The report states that there are three possible explanations for these results. The true reason may never be known as, but we do realize that using substantiated child abuse and neglect reports as a measure for program success is fraught with difficulties. State child abuse data often are considered to be the primary indicator of successful prevention efforts. However, if one wishes to obtain an accurate appraisal of family functioning, using CPS data may not be adequate. For example, a national study found that one third of child maltreatment incidents go unreported (Sedlak & Broadhurst, 1996), so reliance on this data will produce an incomplete picture. Observations of interactions between a parent and child, maternal warmth, sensitivity, and nurturing are better indicators of parent-child relationships than maltreatment reports. Healthy Families Arizona independent evaluations have time and again over the years showed excellent outcomes in these meaningful areas. The Department will cooperate in any way possible in helping to find or develop a better method of determining the impact of the program’s positive outcomes on the prevention of child abuse and neglect. There are no recommendations under this finding. DEPARTMENT OF ECONOMIC SECURITY RESPONSE TO THE HEALTHY FAMILIES ARIZONA PROGRAM EVALUATION FINDING II: Healthy Families Program Provides Health Referrals, Encourages On-Time Immunizations and Promotes Safety The Department agrees the Healthy Families Arizona Program provides health referrals, encourages on-time immunizations and promotes safety in the program’s participant families. The Department is proud of the many successes in these areas. An example of one success is how the direct involvement of a Healthy Families home visitor saved the life of a family of nine. An early morning fire swept through the two-bedroom apartment of a participant family. The mother was alerted in time because the Healthy Families home visitor had recently completed the Child Safety Checklist with the family and had helped them purchase and install smoke detectors in the apartment. The finding of the Auditor General is agreed to and the audit recommendation will be implemented which states that to ensure compliance with its statutorily defined goal to reduce substance abuse, the Healthy Families Program should continue to implement the new questionnaire for assessing participant substance abuse problems and method for tracking substance abuse referrals. The Healthy Families Program started using the new substance abuse assessment questionnaire on July 1, 2000 to measure the effect of the program on reducing dependency on alcohol and drugs over time. Questions have been added to the family update form to enable staff to track referrals for substance abuse treatment. The finding of the Auditor General is agreed to and the audit recommendation will be implemented which states that to ensure families with children who receive scores in the delayed range on the Ages & Stages Questionnaire (ASQ) receive appropriate referrals, the Healthy Families Program should continue to implement new measures for tracking these referrals. As this report states, the Healthy Families Program already revised the family update form, which was implemented on July 1, 2000. The form includes tracking reporting information on referrals for children identified as having a potential developmental delay. This new form contains questions which supplement the ASQ, and is being used to track whether children with developmental delays are offered appropriate referrals. DEPARTMENT OF ECONOMIC SECURITY RESPONSE TO THE HEALTHY FAMILIES ARIZONA PROGRAM EVALUATION FINDING III: Parenting Stress Measure Shows Improved Family Relations The Department agrees the Healthy Families Program has improved family relations in participant families. The outcomes from the Parenting Stress Index show that parental stress decreases after 12 months of participation in the program. As pointed out in this report, research shows that high scores on parenting stress are related to the potential to abuse one’s child, and, conversely, low to normal scores are related to positive parent-child relationships. The program also assists participants with higher stress scores to seek mental health services, thus providing help to parents so that they have a better opportunity to become nurturing parents to their children. The Department fully intends to continue independent evaluation of the Healthy Families Arizona Program and to maintain national accreditation to ensure continued excellence in program delivery and administration. There are no recommendations under this finding. Other Performance Audit Reports Issued Within the Last 12 Months 00-4 Family Builders Pilot Program 00-5 Arizona Department of Agriculture— Licensing Functions 00-6 Board of Medical Student Loans 00-7 Department of Public Safety— Aviation Section 00-8 Arizona Department of Agriculture— Animal Disease, Ownership and Welfare Protection Program 00-9 Arizona Naturopathic Physicians Board of Medical Examiners 00-10 Arizona Department of Agriculture— Food Safety and Quality Assurance Program and Non-Food Product Quality Assurance Program 00-11 Arizona Office of Tourism 00-12 Department of Public Safety— Scientific Analysis Bureau 00-13 Arizona Department of Agriculture Pest Exclusion and Management Program 00-14 Arizona Department of Agriculture State Agricultural Laboratory 00-15 Arizona Department of Agriculture— Commodity Development Program 00-16 Arizona Department of Agriculture— Pesticide Compliance and Worker Safety Program 00-17 Arizona Department of Agriculture— Sunset Factors 00-18 Arizona State Boxing Commission 00-19 Department of Economic Security— Division of Developmental Disabilities 00-20 Department of Corrections— Security Operations 00-21 Universities—Funding Study 00-22 Annual Evaluation—Arizona’s Family Literacy Program 01-01 Department of Economic Security— Child Support Enforcement Future Performance Audit Reports Department of Public Safety—Telecommunications Department of Public Safety—Drug Abuse Resistance Education (D.A.R.E.) Program Board of Osteopathic Examiners in Medicine and Surgery
Object Description
TITLE | Annual evaluation, Healthy Families Program |
CREATOR | Office of the Auditor General |
SUBJECT | Healthy Families Arizona (Program); Arizona--Department of Economic Security--Division of Children and Family Services--Auditing; Families--Health and hygiene--Arizona; |
Browse Topic |
Government and politics Family and community Health & Well-being |
DESCRIPTION | This title contains one or more publications |
Language | English |
Publisher | Office of the Auditor General |
Material Collection | State Documents |
Acquisition Note | Report No. 01-02 |
Source Identifier | LG 6.3:H 31 |
Location | o46943471 |
REPOSITORY | Arizona State Library, Archives and Public Records--Law and Research Library |
Description
TITLE | Annual evaluation, Healthy Families Program |
DESCRIPTION | 66 pages (PDF version). File size: 363 KB |
TYPE |
Text |
Acquisition Note | Report No. 01-02 |
RIGHTS MANAGEMENT | Copyright to this resource is held by the creating agency and is provided here for educational purposes only. It may not be downloaded, reproduced or distributed in any format without written permission of the creating agency. Any attempt to circumvent the access controls placed on this file is a violation of United States and international copyright laws, and is subject to criminal prosecution. |
DATE ORIGINAL | 2001-02 |
Time Period |
2000s (2000-2009) |
ORIGINAL FORMAT | Born Digital |
Source Identifier | LG 6.3:H 31 |
Location | o46943471 |
DIGITAL IDENTIFIER | 01-02.pdf |
DIGITAL FORMAT | PDF (Portable Document Format) |
REPOSITORY | Arizona State Library, Archives and Public Records--Law and Research Library. |
File Size | 371084 Bytes |
Full Text | State of Arizona Office of the Auditor General PERFORMANCE AUDIT Report to the Arizona Legislature By Debra K. Davenport Auditor General HEALTHY FAMILIES PROGRAM February 2001 Report No. 01-02 The Auditor General is appointed by the Joint Legislative Audit Committee, a bipartisan committee composed of five senators and five representatives. Her mission is to provide independent and impar-tial information and specific recommendations to improve the operations of state and local government entities. To this end, she provides financial audits and accounting services to the state and political subdivisions and performance audits of state agencies and the programs they administer. Audit Staff Carol Cullen—Manager and Contact Person (602) 553-0333 Joanne Dukeshire—Team Leader Laurie Cohen—Team Member Copies of the Auditor General’s reports are free. You may request them by contacting us at: Office of the Auditor General 2910 N. 44th Street, Suite 410 Phoenix, AZ 85018 (602) 553-0333 Additionally, many of our reports can be found in electronic format at: www.auditorgen.state.az.us 2910 NORTH 44th STREET • SUITE 410 • PHOENIX, ARIZONA 85018 • (602) 553-0333 • FAX (602) 553-0051 DEBRA K. DAVENPORT, CPA AUDITOR GENERAL STATE OF ARIZONA OFFICE OF THE AUDITOR GENERAL February 15, 2001 Members of the Arizona Legislature The Honorable Jane Dee Hull, Governor Mr. John Clayton, Director Department of Economic Security Transmitted herewith is a report of the Auditor General, an evaluation of the Healthy families Program. This evaluation was conducted pursuant to A.R.S. §41-1279.08. I am also transmitting with this report a copy of the Report Highlights for this evaluation to provide a quick summary for your convenience. As outlined in its response, the Department agrees with all of the findings and recommendations. My staff and I will be pleased to discuss or clarify items in the report. This report will be released to the public on February 16, 2001. Sincerely, Debbie Davenport Auditor General Enclosure i OFFICE OF THE AUDITOR GENERAL SUMMARY The Office of the Auditor General has completed an evaluation of the Healthy Families Program. The Office of the Auditor Gen-eral is required pursuant to Arizona Revised Statutes (A.R.S.) §41-1279.08 to evaluate the Healthy Families Program annually. This report contains information related to program procedures and effectiveness. Healthy Families—Arizona is a child abuse prevention and child wellness and development program administered by the De-partment of Economic Security (DES). The Healthy Families Program has five goals: (1) reduce child abuse and neglect; (2) promote child wellness and proper development; (3) strengthen family relations; (4) promote family unity; and (5) reduce de-pendency on drugs and alcohol. The program coordinates with hospitals to identify mothers giving birth in specific geographic areas whose family character-istics may place them at risk for committing child abuse or ne-glect, and/or whose babies may face increased health risk. Healthy Families provides services through various contractors, which include medical centers, local health departments, and social service agencies. These contractors provide services at 23 sites in 10 counties. The program is based on a home-visitation model. Home visitors regularly visit families to offer support, educational materials, and referrals to needed resources. Home visitors also help fami-lies develop family support plans and encourage positive parent-child interaction to strengthen bonds and promote development. Participation in the program is voluntary and may continue for up to five years. In fiscal year 2000, 1,254 families were enrolled in the program. Summary ii OFFICE OF THE AUDITOR GENERAL Healthy Families Participant Child Abuse Rates Do Not Differ Significantly from the Comparison Group (See pages 13 through 19) For the second year in a row, comparison of child abuse rates for Healthy Families participants and a comparison group of fami-lies who left the program shows that abuse rates were not sig-nificantly different. Specifically, 1.7 percent of participant fami-lies received a substantiated report of abuse and 1.8 percent of families who left the program before receiving four home visits received a substantiated report. These rates are comparable to the rates reported in the previous Office of the Auditor General evaluation of the program (see Report No. 00-1), in which 1.6 percent of participants and 1.4 percent of the comparison group received substantiated reports of abuse. While one possible explanation for these findings might be that the program does not have a significant effect on child abuse, other explanations may also account for the similarity in out-comes between the two groups. First, it is possible that the effects of the program on preventing child abuse cannot be adequately measured by the current evaluation design because of changes in DES procedures for substantiating child abuse reports. In par-ticular, the occurrence of substantiated abuse reports has de-creased since the implementation of an appeals process. Second, preliminary analysis of when abuse occurs for both program participants and the comparison group suggests that abuse is likely to occur a year or more after enrollment, and in most cases, after the family has left the program. Thus, participants may not be enrolled in the program at the time when the risk of child abuse increases. Healthy Families Program Provides Health Referrals, Encourages On-Time Immunizations, and Promotes Safety (See pages 21 through 26) The Healthy Families Program has been successful in providing health referrals, encouraging parents to get their children immu-nized, and in promoting home safety. On average, the majority Summary iii OFFICE OF THE AUDITOR GENERAL of children in the program are fully immunized and most chil-dren have a primary health care provider. Additionally, Healthy Families staff regularly conduct assessments of the physical and social development of children enrolled in the program. Parents improved their compliance with home safety measures such as covering electrical outlets and keeping poisons and choking hazards out of reach. Parenting Stress Measure Shows Improved Family Relations (See pages 27 through 29) Healthy Families parents showed significant decreases in indica-tors of parental stress after one year in the program, suggesting that program services may improve parent-child relations. In addition, parents with higher stress levels related to areas such as social isolation and depression also received referrals to mental health services at a higher rate than parents with lower stress. To assist families in improving parent-child interaction, home visi-tors regularly provide emotional support and information to help parents understand their child’s behavior. Other Pertinent Information (See pages 31 through 34) The previous report issued by the Office of the Auditor General (see Report No. 00-1) offered several recommendations to im-prove the measurement of the program’s effect on family func-tioning and reduction of substance abuse. The previous evalua-tion also recommends tracking referrals for children with devel-opmental delays. To address these recommendations, the pro-gram discontinued using the Home Observation for Measure-ment of the Environment (HOME) questionnaire and will in-stead use the Parenting Stress Index (PSI) to measure the pro-gram’s effect on family relations. The program also replaced the CAGE substance abuse screen with a new questionnaire that can measure the program’s effect on reducing drug/alcohol depend-ency over time. Finally, the program added questions to its fam-ily update form, which will enable staff to track whether children with developmental delays are offered the appropriate referrals. Summary iv OFFICE OF THE AUDITOR GENERAL Statutory Annual Evaluation Components (See pages 35 through 44) This report also contains information that addresses various components of the Healthy Families evaluation statute. These components include information regarding participant demo-graphics such as mother’s age, marital status, income, ethnicity, and household size. Other areas covered in the statutory compo-nents section contain information about enrollment and disen-rollment and employment. v OFFICE OF THE AUDITOR GENERAL TABLE OF CONTENTS Page Introduction and Background.......................... 1 Finding I: Healthy Families Participant Child Abuse Rates Do Not Differ Significantly from the Comparison Group................................ 13 Child Abuse Rates for Healthy Families Participants and Comparison Group Similar............................................ 13 Possible Explanations of Results......................................................................... 16 Finding II: Healthy Families Program Provides Health Referrals, Encourages On-Time Immunizations, and Promotes Safety ............................................................ 21 Background..................................................................... 21 Participants Receive Developmental Screenings, Referrals, and On-Time Immunizations................................................................ 22 Participants Show Significant Improvements in Home Safety...................................... 24 Recommendations .......................................................... 26 Finding III: Parenting Stress Measure Shows Improved Family Relations............. 27 Background..................................................................... 27 Participants’ Stress Reduced After 12 Months.............................................................. 27 Table of Contents vi OFFICE OF THE AUDITOR GENERAL TABLE OF CONTENTS (Cont’d) Page Finding III: (Concl’d) Parents with Higher Stress More Likely to Receive Mental Health Referrals .............................................................. 29 Home Visitors Provide Information on Parent-Child Bonding..................................................... 29 Other Pertinent Information ............................. 31 Statutory Annual Evaluation Components...... 35 Appendix ........................................................... a-i Agency Response Figures Figure 1 Healthy Families Program Contract Provider Locations Year Ended June 30, 2000............................. 8 Figure 2 Healthy Families Program Number of Participants and Comparison Group Members by County January 1994 through May 2000.................. 37 Figure 3 Healthy Families Program Reasons for Exit January 1994 through May 2000.................. 40 Figure 4 Healthy Families Program Employment Rates of Mothers and Fathers in Program January 1994 through May 2000................. 44 Table of Contents vii OFFICE OF THE AUDITOR GENERAL TABLE OF CONTENTS (Cont’d) Page Tables Table 1 Healthy Families Program Schedule of Distributions by Funding Source and Contractor, and Average State Contribution per Family Year Ended June 30, 2000 (Unaudited)................................................... 7 Table 2 Healthy Families Program Percentage of Participants and Comparison Group Members with Substantiated and Unsubstantiated Reports of Abuse and Referrals to Family Builders Reports Received Between January 1, 1998 and May 31, 2000................ 15 Table 3 Healthy Families Program Percentage of Participants and Comparison Group Members with Substantiated Abuse Reports Office of the Auditor General Evaluations.................................................... 16 Table 4 Healthy Families Program Ages and Stages Questionnaire Percentage of Children Developing at Age-Appropriate Levels January 1994 to May 2000............................. 24 Table 5 Healthy Families Program Percentage of Safety Measures Implemented by Age of Child July 1998 to May 2000................................... 25 Table 6 Healthy Families Program Average Parenting Stress Scores After 3 Weeks and 12 Months in the Program July 1998 to May 2000................................... 28 Table of Contents viii OFFICE OF THE AUDITOR GENERAL TABLE OF CONTENTS (Concl’d) Page Tables (Concl’d) Table 7 Healthy Families Program Participant and Comparison Group Member Profile January 1994 through May 2000.................. 36 Table 8 Healthy Families Program Participants and Comparison Group Members Profile of Family Characteristics January 1994 through May 2000.................. 38 Table 9 Healthy Families Program Months of Participation for Families at Exit January 1994 through May 2000.................. 40 1 OFFICE OF THE AUDITOR GENERAL INTRODUCTION AND BACKGROUND The Office of the Auditor General has completed an evaluation of the Healthy Families Program. According to Arizona Revised Statutes (A.R.S.) §41-1279.08, the Office of the Auditor General is required to evaluate the Healthy Families Program annually. This report contains information regarding the program's proce-dures and outcomes. Healthy Families Offers a Preventative Approach to the Problem of Child Abuse Healthy Families—Arizona is a child abuse prevention and child wellness and development program administered by the De-partment of Economic Security (DES). Preventing child abuse is a serious concern in Arizona, as in other states. During fiscal year 1999, Arizona's Department of Economic Security, Division of Child Protective Services received 32,631 reports of maltreatment toward children. Twenty-five percent of these reports were sub-sequently substantiated.1 The State Legislature established Healthy Families—Arizona as a pilot program in 1994, and gave it permanent program status in 1998. Healthy Families attempts to identify families at risk for committing child abuse or neglect and that may have poor health outcomes. The program provides participants with sup-port, education about child development and referrals for nutri-tion, medical care, counseling, and education and employment programs. Arizona's program is part of a national initiative pro-moted by Prevent Child Abuse America. It is currently offered in 39 states, the District of Columbia, and Canada and serves over 400 communities. Program enrollment and participation are 1 Department of Economic Security. Child Welfare Reporting Requirements: Annual Report for the Period of July 1, 1998 through June 30, 1999. Substanti-ated reports are those reports DES determines met the standard of prob-able cause to conclude that abuse occurred. Note: Of the total reports for fiscal year 1999, 153 fell within the jurisdiction of military or tribal gov-ernments. Healthy Families is a child abuse prevention and child wellness and development program. Introduction and Background 2 OFFICE OF THE AUDITOR GENERAL voluntary. However, nationally, 90 percent of families offered Healthy Families services enroll in the program. The Healthy Families Program is based on home visitation. Home visitors regularly visit families to offer support, educa-tional materials, and referrals to various medical, mental health, and social services. (For information regarding home visitor training see the Training, Quality Assurance, and Credentialing section on page 5.) During the home visit, home visitors help families develop family support plans in which parents identify goals for themselves and their children and encourage positive parent-child interaction to strengthen bonds and promote devel-opment. Home visitors also determine which services a family might need, and provide educational material on child health, developmental milestones, safety, discipline, and nutrition. In addition to home visits, the program holds parent and play group meetings to provide participants with the opportunity to meet other families in their community. Families may participate in the program for five years. Program Stresses Early Screening and Intervention Healthy Families establishes its family visitation within the first three months after a child is born. By doing so, the program at-tempts to help families establish positive parent-child relation-ships and positive child development outcomes early on. To identify families most at risk, Healthy Families coordinates with hospitals to screen mothers giving birth within the program's service area (see Figure 1, footnote 1, page 8). To be eligible for program enrollment, participants cannot have any substantiated reports of child abuse on file with Child Protective Services (CPS) and must reside in a Healthy Families service area. The intake process is completed in two stages. Specifically, n The first stage includes a screening in which parents are iden-tified as possibly at risk for child abuse based on a combina-tion of factors such as inadequate income, unstable housing, lack of a high school diploma, inconsistent or late prenatal care, and being unmarried or separated. The screening proc-ess is usually completed while the mother is still in the hospi-tal. The program refers to those screenings that identify at- The program provides services through home visitation. Participants are enrolled within the first three months after their child is born. Introduction and Background 3 OFFICE OF THE AUDITOR GENERAL risk parents as positive screens. During fiscal year 2000, Healthy Families screened 17,005 families. Of this total, 49 percent (8,382) screened positive. n During the second stage, parents who receive a positive screen are contacted by a Healthy Families worker about completing an assessment. If the parent consents to complet-ing the assessment, he or she is interviewed about topics in-cluding family history, history of substance abuse or criminal activity, stress, self-esteem issues, expectations for the child, and plans regarding discipline. If the parent receives a posi-tive assessment, indicating a potential risk for committing child abuse, he or she is invited to enroll in the program pending a CPS check. Assessments are completed a few days after the screening process either in the hospital, in the par-ent's home, or over the telephone. Families who received a positive screen and were not offered the assessment include those who were ineligible for the pro-gram because they have a substantiated abuse report on file with CPS, could not be reached, or lived in an area in which the Healthy Families Program was full. During fiscal year 2000, of those families contacted and offered the assessment, 1,069 completed the assessment and 533 refused the assess-ment. Nine-hundred thirty-one families assessed positive for child abuse risk and 854 of these families enrolled in the pro-gram. Once enrolled in the program, Healthy Families participants are required by statute to perform community service in exchange for program services. The program has established that families must complete 12 hours of community service per year of par-ticipation. Home visitors at each site are responsible for serving 12 to 25 families. The frequency of home visits varies according to the family's level of participation. Visits are more frequent at the beginning of the family's participation and less frequent as the family becomes more self-sufficient. Increased self-sufficiency is determined by a variety of factors such as a stable home envi-ronment, ability to access resources independently, and utiliza-tion of support networks. Introduction and Background 4 OFFICE OF THE AUDITOR GENERAL Program Goals Five program goals are outlined in A.R.S. §8-701. Healthy Fami-lies goals and the procedures used by the program to address these goals are listed below: n Goal 1: Reduce child abuse and neglect—To reduce child abuse and neglect, home visitors assist families with enhanc-ing parent-child interaction, attachment, and bonding. They provide information about child safety, discipline, and meth-ods for anger management, and assess parent stress levels and home safety. As appropriate, home visitors may provide referrals for counseling and treatment services to address substance abuse and domestic violence. n Goal 2: Promote child wellness and proper develop-ment— Home visitors provide information on child health and nutrition and encourage families to have a primary care physician, to receive well-baby check-ups, and to have their children immunized. They also conduct periodic develop-mental assessments and provide referrals to medical provid-ers and the Arizona Early Intervention Program (AzEIP) for children with potential delays. n Goals 3 and 4: Strengthen family relations and promote family unity—Healthy Families uses an approach that fo-cuses services on the family and building on family strengths. They encourage positive parent-child interaction and discuss family relationship issues with participants. When appropri-ate, they refer participants to counselors to discuss family re-lationship issues. Although most primary caregivers who en-roll in the program are mothers, the program encourages fa-thers and other family members to participate in home visits and group activities. n Goal 5: Reduce dependency on drugs and alcohol—To address issues of substance abuse, Healthy Families began administering a substance abuse assessment instrument in July 1998. When alcohol or drug abuse problems are identi-fied, home visitors are trained to discuss the issue with the participant and/or provide a referral for treatment. Introduction and Background 5 OFFICE OF THE AUDITOR GENERAL Training, Quality Assurance, and Credentialing Home visitors are trained home visiting specialists. They must have at least a high school diploma to be employed by the pro-gram, but many also have college degrees. Home visitors receive a minimum of 30 hours of training a year in subjects such as child development, substance abuse, and identifying and report-ing child abuse and neglect. New home visitors are required to complete four and one-half days of initial training. New staff also complete self-guided training modules under the supervision of site managers. Initial training includes methods for identifying child abuse and neglect, infant growth and development, and methods for interacting with families and encouraging positive parent-child interaction. Finally, new staff must also complete on-the-job training in which they observe home visits conducted by more experienced staff. To help ensure that Healthy Families’ sites are in compliance with program policies and procedures, DES contracts with an evaluation firm to provide quality assurance and database man-agement. Specifically, quality assurance coordinators from this firm visit each site at least twice a year to review participant files for accuracy. This firm also collects and maintains all program data from each Healthy Families site and is responsible for test-ing this data to ensure that it is reliable and accurate. In April 2000, Healthy Families—Arizona became the first pro-gram in the nation to receive a statewide credential from Prevent Child Abuse America and the Council on Accreditation. During the credentialing process, the program had to demonstrate ad-herence to national standards by submitting information on training, technical assistance, policies, quality assurance, evalua-tion, and administration. Each site completed a self-assessment of its performance according to 12 critical elements and 138 stan-dards of best practice. Seven sites received formal visits from national credentialing committee representatives. During the site visits, committee representatives evaluated the quality of service provided by each site. The credential attests that Healthy Families—Arizona meets nationally established research-based standards for quality ser- Healthy Families—Arizona was first in the nation to re-ceive a statewide credential. Introduction and Background 6 OFFICE OF THE AUDITOR GENERAL vice delivery, and best practice standards for management and administration. Contractors and Appropriations Healthy Families provides services through various contractors, which include medical centers, local health departments, and social service agencies. In fiscal year 2000, DES had 11 contracts with 10 separate contractors. These contractors provide services at 23 sites in 10 counties (see Figure 1, page 8). For fiscal year 2000, two new sites were added in Maricopa County. In addi-tion, the Pascua Yaqui Tribe site offered services in previous years through a federal grant but was awarded program funding for the first time this year. Each site serves selected areas within a 40-mile radius of the contractor's office. These areas, identified by their zip codes, are chosen based on the number of live births per year, the number of CPS reports for children ages 0 to 5 years, and other factors, including low income and under-utilization of health care services. Additionally, Healthy Families contracts with an evaluation firm for database management, evaluation, quality assurance, and training. For fiscal year 2000, program funding for Healthy Families to-taled about $5.6 million, $4 million of which was from the State’s General Fund appropriation. In fiscal years 1997, 1998, and 1999 the General Fund appropriation totaled about $3 million for each fiscal year. Other funding sources for the program include DES' Child Abuse Prevention Fund, federal grants, contractor contri-butions, and private donations (see Table 1, page 7). Scope and Methodology A.R.S. §41-1279.08 mandates that the Office of the Auditor Gen-eral evaluate the Healthy Families Program. The statute specifies that the evaluation will include an assessment of the program's effectiveness in achieving its goals. According to the statute, additional evaluation requirements include providing informa-tion about the level and scope of program services, enrollment eligibility, participant demographic characteristics, long-term Healthy Families provides services through local contrac-tors. Introduction and Background 7 OFFICE OF THE AUDITOR GENERAL Table 1 Healthy Families Program Schedule of Distributions by Funding Source and Contractor, and Average State Contribution per Family 1 Year Ended June 30, 2000 (Unaudited) Funding Source Contractor: State General Fund Child Abuse Prevention Fund Tobacco Taxes 2 Federal Grants 3 Total Southwest Human Development $1,098,787 $179,892 $132,580 $195,837 $1,607,096 Child and Family Resources: Urban 769,980 106,530 213,923 1,090,433 Rural 539,699 184,819 183,569 908,087 Coconino County Department of Pub-lic Health 296,761 33,506 31,586 1,325 363,178 Yavapai Regional Medical Center 215,587 22,773 25,534 263,894 Pinal County Department of Public Health 171,605 45,512 24,735 241,852 Lake Havasu Social Services Inter-agency Council 190,772 20,470 20,470 231,712 Yuma County EXCEL Group 108,903 40,743 20,008 169,654 Verde Valley Medical Center 88,875 30,415 16,590 135,880 Pascua Yaqui Tribe 8,254 38,730 27,792 74,776 Parents Anonymous of Arizona (Tuba City) 32,940 20,790 10,530 64,260 LeCroy & Milligan (provides evalua-tion, quality assurance, and training statewide) 445,043 36,832 481,875 Total monies distributed $3,967,206 $ 724,180 $ 716,357 $224,954 $5,632,697 Average state contribution per family 4 $ 3,164 $ 577 $ 571 $ 4,312 1 Department-distributed monies to contractor, rather than contractor expenditures, are presented. Contractors do not report expenditures in a timely manner and funding sources appear to be approximate expenditures of Department-distributed monies. In addition, the schedule excludes the required contractor matches of at least 10 percent, because con-tractors do not report the value of their noncash resources, such as office space and personnel, on a consistent basis. 2 The program received monies from the Parent’s Commission on Drug Education and Prevention beginning in January 2000. The Commission’s funding comes from tobacco taxes; therefore, this schedule presents the source of funding as to-bacco taxes. 3 Consists of approximately $164,400 and $60,600 received from the Community Based Family Resource and Support and the Safe and Stable Families Act federal grants, respectively. 4 Calculation based on the total number of families served (1,254) during the fiscal year, including families who have disenrolled. Calculation excludes the federal grants and contractor contributions to arrive at the State’s contribution per family. Source: Auditor General staff analysis of financial information provided by the Department of Economic Security. Introduction and Background 8 OFFICE OF THE AUDITOR GENERAL Figure 1 Healthy Families Program Contract Provider Locations1 Year Ended June 30, 2000 u Coconino County Health Serving Page and Flagstaff areas u Verde Valley Medical Center Serving Verde Valley area u Yavapai Regional Medical Center Serving Prescott area u Inter-Agency Council Serving Lake Havasu and Parker areas u Southwest Human Development Serving 7 sites in Maricopa County u Excel Group Serving Yuma area u Pinal County Health Serving Casa Grande and Coolidge areas Child & Familyuu Resources Serving 4 sites in Metro Tucson u u Child & Family Resources Serving Nogales Sierra Vista, and Douglas/Bisbee areas u 1 Each site serves selected areas within a 40-mile radius of the contractor’s office. These areas, identified by their zip codes, are chosen based on the number of live births per year, the number of CPS reports for children ages 0 to 5 years, and other factors including low income and under-utilization of health care services. Source: Auditor General staff analysis of Department of Economic Security Healthy Families contracts for the year ended June 30, 2000. Pascua Yaqui Tribe Parents Anonymous Serving Tuba City u u ¿ Introduction and Background 9 OFFICE OF THE AUDITOR GENERAL savings associated with the program and rates of enrollment and disenrollment. During this annual evaluation, Auditor General staff visited 11 of 23 sites. Site visits included: n An interview with site manager(s); n A group interview with home visitors and assessment work-ers; n Accompanying home visitors on 1 to 2 home visits. During home visits, evaluators interviewed 13 program participants and conducted 13 structured observations of the visit; and n Reviews of 110 participant files to check the accuracy of Healthy Families databases. Specifically, information ob-tained from the file reviews was compared to information in the program's database. This comparison showed that data was reliable. Evaluators also analyzed data from the following assessment tools: n The Parenting Stress Index, which provides a measure of parental stress; n The Child Safety Checklist, which measures the safety of the home environment; n The Ages and Stages Questionnaire, which assesses children for potential developmental delays. A description of the assessment tools is included in the Appen-dix (see pages a-i through a-ii). In addition to the assessment tools, evaluators collected and analyzed: n Approximately 83,000 records of substantiated and unsub-stantiated CPS reports to determine the number of reports re-ceived by program participants and members of the com-parison group; Introduction and Background 10 OFFICE OF THE AUDITOR GENERAL n Immunization rates of children in the program; n Healthy Families participant usage of two types of public assistance: Temporary Assistance for Needy Families (TANF) and food stamps. In conducting the analyses of child abuse rates, evaluators com-pared substantiated and unsubstantiated report rates for Healthy Families participants with those of a comparison group. Healthy Families participants included in the analysis of abuse rates are those participants who enrolled in the program from July 1, 1997 to November 30, 1999, received at least four home visits, and participated in the program for at least six months. The compari-son group comprised individuals who enrolled in Healthy Fami-lies between July 1, 1997 and November 30, 1999, but left the program before receiving four home visits. Evaluators also ana-lyzed CPS data to determine how many participants and com-parison group members were referred to Family Builders, which is a program that provides services to families who have been reported to CPS for low and potential risk of child abuse. The evaluation includes findings and recommendations in the following areas: n Child abuse rates for Healthy Families participants and the comparison group are not significantly different; n The Healthy Families program succeeds in providing health referrals, encouraging on-time immunizations, and promo-ting safety. n Parenting stress measure shows improved family relations. In addition to these finding areas, the evaluation also contains an Other Pertinent Information section (see pages 31 through 34), regarding the implementation of new methods for measuring program outcomes. These changes address recommendations made in the last Auditor General Report (Report No. 00-1). Introduction and Background 11 OFFICE OF THE AUDITOR GENERAL Acknowledgements The Auditor General and staff express appreciation to the Direc-tor of the Department of Economic Security and DES staff in the Division of Children, Youth, and Families and the Office of Evaluation; the Manager of the Office of Prevention and Family Support, Healthy Families Statewide Coordinator, and Program Specialist; Healthy Families Data Management, Evaluation and Quality Assurance staff; Healthy Families supervisors, home visitors, and family assessment workers; and the Department of Health Services' Bureau of Epidemiology and Disease Control for their cooperation and assistance during this evaluation. 12 OFFICE OF THE AUDITOR GENERAL (This Page Intentionally Left Blank) 13 OFFICE OF THE AUDITOR GENERAL FINDING I HEALTHY FAMILIES PARTICIPANT CHILD ABUSE RATES DO NOT DIFFER SIGNIFICANTLY FROM THE COMPARISON GROUP For the second year in a row, comparison of child abuse rates for Healthy Families participants and families who left the program show that abuse rates were not significantly different. While one possible explanation of these findings might be that the program does not have a significant effect on child abuse, other explana-tions may also account for the similarity in outcomes between the two groups. Child Abuse Rates for Healthy Families Participants and Comparison Group Similar Child abuse rates for families who participated in the Healthy Families Program did not differ significantly from child abuse rates for a comparison group of families who left the program after only a short time. Additionally, substantiated reports for many participants and control group members occurred a year or more after program enrollment. Finally, abuse rates found in this report are consistent with rates found in the Auditor Gen-eral’s last evaluation (see Report No. 00-1). Evaluation compared Healthy Families participants with others who left the program—To assess the program’s effect on child abuse, evaluators reviewed CPS records in the Children’s Infor-mation Library Data Source (CHILDS).1,2 The analysis included 1 The data system contains abuse report information such as the alleged abuser, report date, reporting source, and investigation findings. 2 Auditor General staff also reviewed the CHILDS system’s intake, investi-gation, and security procedures to determine whether adequate controls for ensuring the reliability and accuracy of data were in place. Staff con-cluded that such controls were in place. Finding I 14 OFFICE OF THE AUDITOR GENERAL all substantiated and unsubstantiated CPS reports made at least six months after intake in Healthy Families for participants and comparison group members. This time period was selected to allow participants to have sufficient program exposure before assessing whether the program had an impact. For instance, for families with intake dates of July 1, 1997, evaluators examined reports made on or after January 1, 1998. Evaluators also exam-ined the proportion of participants and comparison group mem-bers who received referrals to Family Builders during the same period. The Healthy Families participant group comprised families who received at least four home visits and participated in the pro-gram for at least six months; members of the comparison group left the program before receiving four home visits. The analysis included 1,139 program participants and 512 families in the comparison group. No significant difference between Healthy Families and com-parison group—The analysis revealed that Healthy Families participants and comparison group members had similar rates of CPS reports. For substantiated reports—those in which DES concluded that abuse occurred—the percentage of families who had a report on file was 1.7 for program participants, and 1.8 percent for the comparison group. The slight difference between the two is not statistically significant. For unsubstantiated re-ports, program participants had a slightly higher rate (6.1 percent versus 5.5 percent for the comparison group), but again the dif-ference was not statistically significant. As a further point of comparison, evaluators also analyzed the percentage of program participants and comparison group members who were referred to Family Builders. Family Builders is a program that provides services to families who have been reported to CPS for low and potential abuse risk. Once referred to Family Builders, the report is closed and CPS does not conduct an investigation of the case.1 Results show that 1.8 percent of program participants and 2.7 percent of the comparison group 1 To be eligible for Family Builders, the family cannot have an existing open file with CPS. Additionally, the case cannot involve allegations of sexual abuse, or current injuries, or involve a child who is currently in out-of-home placement or a ward of the State. Participant abuse rates do not differ from comparison group. Finding I 15 OFFICE OF THE AUDITOR GENERAL were referred to Family Builders. However, this difference is also not statistically significant. Results are summarized in Table 2. Table 2 Healthy Families Program Percentage of Participants and Comparison Group Members with Substantiated and Unsubstantiated Reports of Abuse and Referrals to Family Builders Reports Received Between January 1, 1998 and May 31, 2000 Outcome Measure Program Participants Comparison Group Substantiated reports of abuse 1 1.7% 1.8% Unsubstantiated reports of abuse 1 6.1 5.5 Referred to Family Builders 1 1.8 2.7 1 Differences are not statistically significant. Source: Auditor General staff analysis of data provided by Child Protective Services and the Healthy Families Program. Many reports received one year after intake—The majority of program participants and comparison group families with sub-stantiated reports of abuse received these reports at least one year after their initial intake into the Healthy Families Program. Over half the reports were received two years after intake.1 Moreover, the results of this analysis also show that 79 percent of Healthy Families participants who have a substantiated report in the CHILDS database received the report after they left the pro-gram. Comparison with prior Auditor General reports—The rates found in this report are consistent with those found in the Audi-tor General’s last evaluation (see Report No. 00-1). However, an Auditor General evaluation issued in January 1998 found higher 1 For this analysis, evaluators included all substantiated reports that oc-curred at any point after intake—that is, they did not limit the analysis to reports that occurred after six months of exposure to the program. Finding I 16 OFFICE OF THE AUDITOR GENERAL abuse rates (see Report No. 98-1). Specifically, 3.3 percent of pro-gram participants and 8.5 percent of comparison group members received substantiated reports of abuse (see Table 3). These find-ings show a decrease in rates over the last two years. Possible Explanations of Results There are several possible explanations for the evaluation results. First, it is possible that the effects of the program on preventing child abuse cannot be adequately measured by the current evaluation design because of various factors associated with changes in abuse reporting requirements. Second, it is possible the program’s impact is lessened because most program services occur before the time period when the risk of child abuse in-creases. Third, the program may not have an effect on prevent-ing child abuse. Explanation 1: Program reduces child abuse risk but vari-ous factors might have affected the results—Compared to the 1998 evaluation, Healthy Families participants and compari-son group members were found to have lower rates of abuse in Table 3 Healthy Families Program Percentage of Participants and Comparison Group Members with Substantiated Abuse Reports Office of the Auditor General Evaluations Issue Date Healthy Families Participants Comparison Group January 1998 a 3.3% 8.5% February 2000 b 1.6 1.4 February 2001 b 1.7 1.8 a Difference between Healthy Families participants and comparison group statistically significant at the .001 level. That is, the probability that the difference in rates occurred by chance is less than 1 in 1,000. b Difference between Healthy Families participants and comparison group not statistically significant. Source: Auditor General staff analysis of data provided by Child Protective Services and the Healthy Families Program. There are several possible explanations for evaluation results. Finding I 17 OFFICE OF THE AUDITOR GENERAL the current and last evaluations. The reduction in abuse rates for both groups may reflect changes in CPS child abuse substantia-tion standards, changes in the construction of the comparison group, and changes in program enrollment that occurred after the 1998 evaluation. These changes may have affected the evaluation design, limiting the ability of evaluators to assess the program’s impact on child abuse. Specifically, n Overall there is a lower occurrence of substantiated abuse, due in part to the appeals process—Effective January 1, 1998, DES changed its procedures for substantiat-ing reports of abuse and neglect. Under A.R.S. §8-811, the Department created the Protective Service Review Team (PSRT). The PSRT reviews reports of abuse and/or neglect to determine if report allegations should be substantiated in the Department’s Central Registry.1 Since the implementation of the PSRT, substantiation rates have decreased. According to a DES report, 25 percent of CPS cases were substantiated in fis-cal year 1999, the first full fiscal year with the appeals process in place.2 In comparison, 45 percent of cases were substanti-ated in fiscal year 1997, the last fiscal year without the ap-peals process. n Comparison group different—The current comparison group comprised families who left the program before receiving four home visits. This comparison group differs from the group used in the 1998 evaluation, which was con-structed using individuals eligible for Healthy Families but who could not enroll because the program was full. Al-though current comparison group families did not receive four home visits, many received one to three visits. During these initial visits, Healthy Families staff explain the pro-gram’s mandatory CPS reporting policy and provide infor-mation about child safety (for example, the importance of never shaking a baby, or leaving a baby unattended). Within 1 PSRT reviews do not include those cases in which criminal or civil charges have been filed against the alleged abuser. 2 Department of Economic Security. Child Welfare Reporting Requirement: Annual Reports for the Period July 1, 1996 through June 30, 1997, and the Period of July 1, 1998 through June 30, 1999. Finding I 18 OFFICE OF THE AUDITOR GENERAL the first three visits these families can also receive various re-ferrals for family assistance. In addition, in the current analysis, there are some small but statistically significant differences between participants and the comparison group. Participants are slightly older, less likely to be single, and less likely to be employed. Participant scores on the Family Stress Checklist Risk Assessment are also higher by an average of one point. No significant differ-ences were found when comparing annual household in-come, education level, or ethnicity. n Families with abuse history excluded from the pro-gram— Effective June 1, 1998, the Healthy Families statute (A.R.S. §8-701) was changed, prohibiting the enrollment of families with prior substantiated CPS abuse reports. Families who received a substantiated report, but were already en-rolled in the program, were also required to exit.1 Thus, fami-lies with a demonstrated propensity for abuse are no longer enrolled in the program. Explanation 2: Program impact on abuse may be affected by timing and length of participation—Evaluators’ preliminary analysis of report dates for substantiated abuse reveals that the majority of program participants and comparison group mem-bers who receive reports do so one year or more after the fam-ily’s initial exposure to the program. In most cases, the abuse reports for program participants also occurred after the family left the program as a majority of families participate for 12 or fewer months. Therefore, it is possible that the risk of child abuse may increase as the child gets older, but that participants have already left the program before this increase occurs. 1 Comparison of initial assessment scores for parents who entered the pro-gram before June 1, 1998, with scores of those parents who entered after June 1, 1998, suggest current enrollees are less at-risk. Specifically, Family Stress Checklist (FSC) scores are lower for participants who entered the program after the enrollment criteria changed. The average FSC score is 38.6 for pre-June 1, 1998, enrollees and 36.4 for post-June 1, 1998, enrollees. This difference is significant at the .001 level. Scores of 25 or higher indicate a potential risk to commit abuse. Finding I 19 OFFICE OF THE AUDITOR GENERAL Explanation 3: Groups do not differ because program might not reduce child abuse risk—Because last year’s and this year’s evaluations show no difference in program participant and comparison group abuse rates, it could be concluded that the program does not reduce child abuse risk. However, before making this determination, evaluators would prefer to ad-dress the other possible explanations. This would involve constructing additional comparison groups of “at-risk” fami-lies who have not received any child abuse prevention ser-vices. Additionally, because abuse often occurs a year or more after program enrollment, other measures of the pro-gram's long-term effect will have to be developed. However, establishing new comparison groups and developing addi-tional long-term measures would require extensive efforts and the results of the further analyses would likely not be available for several years. 20 OFFICE OF THE AUDITOR GENERAL (This Page Intentionally Left Blank) 21 OFFICE OF THE AUDITOR GENERAL FINDING II HEALTHY FAMILIES PROGRAM PROVIDES HEALTH REFERRALS, ENCOURAGES ON-TIME IMMUNIZATIONS, AND PROMOTES SAFETY Although the Healthy Families Program’s long-term effect on preventing child abuse cannot be clearly determined, it has been successful in providing health referrals, encouraging on-time immunizations, and promoting safety. More specifically, on average, the majority of children in the program are fully immu-nized and most children have a primary health care provider. Additionally, most children receive regular developmental as-sessments. Parents also improved their compliance with home safety measures over time. Background According to program goals defined in the Healthy Families statute (A.R.S. §8-701), the program is required to provide par-ents with information about child development and preventative health care. Program goals also include reducing participant substance abuse. To determine if the program is achieving these goals, evaluators examined several health, safety, and develop-ment measures. The measurement tools used to collect this in-formation are described in the Appendix (see pages a-i through a-ii). Although program goals include reducing participant substance abuse, evaluators reported in the February 2000 evaluation (see Auditor General Report No. 00-1) that the program’s impact on reducing participants’ dependency on drugs and alcohol could not be determined. Specifically, evaluators reported in the Finding II 22 OFFICE OF THE AUDITOR GENERAL February 2000 evaluation that the CAGE1 substance abuse screen used by the program was not an adequate measure for assessing change in drug and alcohol usage and recommended replacing this instrument. To address this recommendation, the Healthy Families Program began using a new substance abuse question-naire in July 2000 (see Other Pertinent Information, pages 31 through 34). However, data from the questionnaire will not be available for analysis until the 2001 evaluation. Participants Receive Developmental Screenings, Referrals, and On-Time Immunizations Evaluators found that consistent with the January 1998 and Feb-ruary 2000 evaluations, the majority of children in the Healthy Families Program received age-appropriate immunizations and developmental screenings. Most participants also had a primary health care provider. Results of the developmental assessments show that most children in the program are developing nor-mally. However, for those children identified as potentially hav-ing delays, the program provides referrals to appropriate ser-vices and recently began tracking those referrals in its new fam-ily update form (see Other Pertinent Information on page 31). Healthy Families monitors immunization rates and promotes preventative medical care—Eighty-five percent of two- to three-month- old children enrolled in the program are fully immu-nized. At four months of age, 73 percent of children in the pro-gram are fully immunized; at 6 months of age, 66 percent are fully immunized; and at 1 year to 15 months of age, 73 percent are fully immunized. Generally, these rates are higher than esti-mated vaccination rates for children served by local public health clinics in the same areas as program sites. Additionally, these 1 The acronym “CAGE” is derived from the first letter in each of four ques-tions that are asked of the respondent: (1) Have you every felt the need to cut down on drinking/drug use? (2) Have you ever felt annoyed by oth-ers’ criticism of your drinking/drug use? (3) Have you ever felt guilty about your drinking/drug use? (4) Have you ever had a drink/taken drugs first thing in the morning (eye-opener)? Two or more “yes” re-sponses are considered indicative of a substance abuse problem. Finding II 23 OFFICE OF THE AUDITOR GENERAL rates are comparable to statewide rates for Arizona reported by the Centers for Disease Control and Prevention. In addition to encouraging parents to ensure that their children are fully immunized, the program also emphasizes the impor-tance of preventative and routine medical care. At 2 months in the program, 97 percent of children had a medical primary care provider; and at 6 and 12 months, 98 percent of children had a medical provider. To promote positive health outcomes, home visitors ask about immunization schedules for the children they serve, provide educational information about immunizations, and refer families to health/immunization clinics. In addition, program staff em-phasize the importance of routine and preventative medical care and encourage participants to use referrals to obtain a primary care provider. Healthy Families program conducts developmental assess-ments— Home visitors use the Ages and Stages Questionnaire (ASQ) to conduct regular assessments of the physical and social development of children enrolled in the program. The ASQ is administered at the following ages: 4, 6, 12, 18, 24, 30, 36, and 48 months. In addition to conducting these assessments, home visi-tors provide families with information about the developmental stages to alert parents to signs of potential delays and reduce fears associated with unrealistic expectations. Program partici-pants cited developmental assessments as important for identify-ing milestones, early detection of potential delays, and reducing confusion about their child’s development. Results of the ASQ reveal that most children in the program are developing at levels that are appropriate for their age (see Table 4, page 24). The ASQ assesses five developmental areas— communication, gross motor skill, fine motor skill, problem solv-ing, and personal social development. If a child receives a score below the minimum level for normal development, home visi-tors are trained to explain the result to the parent and provide referrals to a primary health care provider, or the Arizona Early Intervention Program (AzEIP), to arrange for further assessment. However, as with the previous Auditor General report (see Re-port No. 00-1), evaluators were unable to directly link develop- Program conducts develop-mental assessments of partici-pants’ children. Finding II 24 OFFICE OF THE AUDITOR GENERAL mental screenings with referrals to AzEIP or a health care pro-vider. The program has revised its family update form. The up-date form now contains questions which enable the program to document if children with developmental delays are offered the appropriate referrals. This change is discussed in the Other Per-tinent Information section of this report (see pages 31 through 34). Table 4 Healthy Families Program Ages and Stages Questionnaire Percentage of Children Developing at Age-Appropriate Levels January 1994 to May 2000 Age in Months Type of Development 4 6 12 18 24 30 36 48 Communication 98.4% 99.8% 98.6% 89.8% 89.4% 91.8% 92.4% 96.7% Gross motor skills 91.7 98.5 97.5 99.2 95.6 95.9 98.2 98.9 Fine motor skills 96.5 99.0 98.8 98.9 96.0 91.2 95.5 98.9 Problem solving 96.7 99.1 96.0 98.6 93.8 93.9 89.2 81.9 Personal/Social 96.5 98.6 97.0 99.9 93.3 94.7 94.6 100.0 Number of children1 1,884 1,849 1,183 741 499 342 224 94 1 The number of children varies by scale; the number reported is the largest number assessed for that age group. Source: Auditor General staff analysis of data provided by the Healthy Families Program. Participants Show Significant Improvements in Home Safety The safety of program participants’ homes improves over the course of their enrollment in the program. Healthy Families staff administer the Child Safety Checklist (CSC) to identify potential safety hazards and also discuss with parents ways to improve home safety. On average, program participants improved their home safety after one year by taking steps to comply with items on the CSC. Finding II 25 OFFICE OF THE AUDITOR GENERAL Healthy Families participants maintain homes that meet child safety requirements—Evaluators analyzed results from the CSC at 2, 6, and 12 months and found that home safety improves after a year of enrollment. On average, the percentage of child safety measures implemented improves from 87 percent to 94 percent between 2 to 12 months (see Table 5). Analyses of individual items on the checklist show significant improvement in key safety measures such as covering outlets and keeping poisons and small items that are potential choking hazards out of reach (see Appendix, page a-ii, for a description of the checklist). Nearly all of the program participants indicated initial (2 months) and subsequent (6 and 12 months) compliance with other important safety measures such as using car seats, making sure their child is never alone in the house or car, and keeping dangerous objects such as scissors and matches out of their child’s reach. Table 5 Healthy Families Program Percentage of Safety Measures Implemented by Age of Child July 1998 to May 2000 Age of Child Percentage Implemented 1 2 months 87% 6 months 91 12 months 94 1 Increases in percentages of safety measures implemented are all statisti-cally significant at the .001 level. That is, the probability that the average gain in compliance occurred by chance is less than 1 in 1,000. Source: Auditor General staff analysis of data provided by the Healthy Fami-lies Program. Participants improve home safety. Finding II 26 OFFICE OF THE AUDITOR GENERAL Recommendations 1. To ensure compliance with its statutorily defined goal to reduce substance abuse, the Healthy Families Program should continue to implement the new questionnaire for as-sessing participant substance abuse problems and method for tracking substance abuse referrals. 2. To ensure that families with children who receive scores in the delayed range on the ASQ receive appropriate referrals, the Healthy Families Program should continue to implement new measures for tracking these referrals. 27 OFFICE OF THE AUDITOR GENERAL FINDING III PARENTING STRESS MEASURE SHOWS IMPROVED FAMILY RELATIONS Results from the Parenting Stress Index show that parental stress decreases after 12 months of participation. Evaluators also found that parents with higher scores on the restricted role, isolation, and depression subscales were more likely to receive referrals to mental health services compared with those whose scores indi-cated lower risk. To assist families in improving parent-child interaction, home visitors provide emotional support and infor-mation to help parents understand their child’s behavior. Background To address recommendations in the last Auditor General evalua-tion, the Healthy Families program discontinued using the Home Observation for Measurement of Environment (HOME) to assess family functioning (see “Other Pertinent Information,” pages 31 through 34). In place of the HOME, the program is using the Parenting Stress Index (PSI) to measure family rela-tions. The PSI is first administered to program participants after 3 weeks of enrollment and then again at 6 and 12 months. Cur-rently, there is no comparison group for this measure. Participants’ Stress Reduced After 12 Months Although scores for most Healthy Families participants are within normal ranges, after 12 months in the program partici-pants show statistically significant decreases in parental stress on 6 of the PSI’s subscales. As shown in Table 6, stress related to attachment, role restriction, competence, social isolation, depres-sion, and mood is lower at 12 months than it is at 3 weeks. Parents have reduced stress after 12 months in the pro-gram. Finding III 28 OFFICE OF THE AUDITOR GENERAL Research has shown that high scores on parenting stress are related to the potential to abuse one’s child. Conversely, low to normal scores are related to positive parent-child relationships. Thus, reductions in parental stress suggest that program services may improve family relations. Table 6 Healthy Families Program Average Parenting Stress Scores After 3 Weeks and 12 Months in the Program July 1998 to May 2000 Participant Scores Scale1 Normal Stress Range 3 Weeks 12 Months Competence2 23-34 31.9 29.9 Attachment2 10-15 13.1 12.0 Role Restriction2 14-23 20.0 18.5 Depression 2 16-24 20.2 19.1 Isolation 2 10-16 14.1 13.2 Distractibility 3 20-28 25.3 25.0 Mood 2 7-11 10.9 9.4 1 Higher numbers indicate higher stress levels. See Other Pertinent Information (pages 31 through 34) for detailed information on these scales. 2 Differences from 3 weeks to 12 months are statistically significant at the .05 level or bet-ter. That is, the chance that the average reduction in parenting stress occurred by chance is less than 5 in 100. Evaluators also analyzed participant PSI scores at 6 months. For the competence, role restriction, and mood subscales evaluators found significant differences between scores at 3 weeks and 6 months. Scores for the depression and attachment sub-scales at 6 months were significantly different from scores at 12 months. 3 Differences not statistically significant. Source: Auditor General staff analysis of data provided by the Healthy Families Program. Finding III 29 OFFICE OF THE AUDITOR GENERAL Parents With Higher Stress More Likely to Receive Mental Health Referrals Additionally, participants who scored in the higher risk ranges for depression, isolation, and role restriction were more likely than families scoring at lower risk levels to receive referrals for mental health services. This shows that the PSI may also be a useful tool for identifying families who need additional services. Home Visitors Provide Information on Parent-Child Bonding To assist families with parent-child interaction, home visitors provide emotional support and information to help parents un-derstand their child’s behavior. Information provided can in-clude methods for alleviating stress, activities to improve parent-child interaction and bonding, and referrals to counseling ser-vices to assist with family relationship problems. 30 OFFICE OF THE AUDITOR GENERAL (This Page Intentionally Left Blank) 31 OFFICE OF THE AUDITOR GENERAL OTHER PERTINENT INFORMATION The previous report issued by the Office of the Auditor General (see Report No. 00-1) offered several recommendations to im-prove the measurement of the Healthy Families Program’s effect on family functioning and reduction of substance abuse. It is necessary for the program to continue collecting data on the program’s effect on family relations and reducing dependency on drugs and alcohol because these are defined in statute (A.R.S. §8-701) as two of the program’s five goals. The previous evalua-tion report also recommends tracking referrals for children with developmental delays. To address these recommendations, the program has implemented several changes. Healthy Families stopped using the HOME—During last year’s evaluation, evaluators were not able to make definitive conclu-sions about the program’s impact on family functioning because the analysis was limited by reliability problems with the Home Observation for Measurement of the Environment (HOME) assessment tool. Specifically, statistical analyses of variety, accep-tance, and organization—three of six areas assessed by the HOME—revealed that questions within each area were not strongly related to each other. This suggests that some items on the HOME do not measure the concepts they were intended to assess. Following the Auditor General’s recommendation, the program stopped using the HOME. However, because improving family relations and family unity are defined in statute as program goals, it is necessary for the program to collect information re-lated to these goals. Therefore, in place of the HOME, the pro-gram will use the Parenting Stress Index (PSI) to assess the pro-gram’s impact on family relations. Past research has shown that low scores on the PSI are related to positive parent-child relation-ships. In particular, the PSI, designed to measure “stressful par-ent- child systems,” can identify areas in which improvements are needed in parent-child relations. Below are descriptions of what high shores on each subscale indicate, according to Program stopped using the HOME assessment. Other Pertinent Information 32 OFFICE OF THE AUDITOR GENERAL Richard Abidin, PSI’s creator1. The first five scales measuring competence, isolation, attachment, restricted role, and depression focus on parent characteristics. The last two scales, measure child distractibility and child mood, focus on child characteristics. n Competence: High scores on the competence scale can be the result of the lack of child development knowledge, lim-ited child management skills, or a feeling that parenting is less reinforcing than expected. High scores may also be re-lated to “lack of acceptance and criticism from the child’s other parent.” n Isolation: Parents who score high on the social isolation scale tend to be “isolated from their peers, relatives, and other emotional support systems.” Additionally, “in many in-stances, their relationships with their spouses are distant and lacking in support for their efforts as parents.” n Attachment: High scores on attachment may indicate that parents do not feel closeness with their child or feel unable to understand the child’s feelings and needs. n Restricted Role: High scores indicate that the parent sees the child as restricting their freedom and may feel controlled by their child’s demands or needs. As a result, resentment and anger may build toward one’s child and/or spouse. n Depression: High scores on the depression scale can be indicative of clinical depression. Depressed parents may lack the energy needed to fulfill parental responsibilities. n Child Distractibility: High scores on this subscale indicate that either the child has Attention Deficit Disorder with hy-peractivity or that the child is normal, but that the parent lacks the energy to keep up with the child. n Child Mood: High scores on child mood are associated with children who are unhappy and depressed. When scores are extremely high, there may be impairment in parental at- 1 Richard R. Abidin. Parenting Stress Index: Professional Manual. Odessa, FL: Psychological Assessment Resources, Inc., 1995. Other Pertinent Information 33 OFFICE OF THE AUDITOR GENERAL tachment or parents may be absent or unavailable due to drug abuse or alcoholism. Healthy Families replaced the CAGE—Evaluators noted in the previous report that, lacking a better instrument, Healthy Fami-lies has used the CAGE substance abuse screening measure since July 1, 1998. However, the CAGE, which contains questions for identifying individuals with an alcohol and/or drug abuse prob-lem, does not allow the program to track whether a participant’s problem with substance abuse changes over time. To address the Auditor General recommendation to replace the CAGE, the pro-gram began using a new questionnaire on July 1, 2000. The new questionnaire measures the program’s effect on reducing de-pendency on alcohol and drugs over time. In addition, the pro-gram has also added questions to its family update form, which will enable staff to track referrals for substance abuse treatment. The new substance abuse assessment was adapted from the Health Screening Survey (Fleming and Barry, 1991) and includes items that measure current substance usage. For instance, par-ticipants are asked to indicate, over the past three months, the number of days per week they drank alcohol, and the number of drinks they typically have had during a single occasion. Unlike the CAGE, which only captures lifetime usage, the new assess-ment allows evaluators to assess changes in drinking patterns over time. The program will track referrals for children with delays—The previous Auditor General report noted that when children re-ceive scores in the delayed range on the Ages and Stages Ques-tionnaire (ASQ) home visitors are trained to provide a referral to a medical provider or the Arizona Early Intervention Program (AzEIP). However, because most families in the program receive medical or social services, evaluators were unable to determine the extent to which families receive services directly related to a developmental delay identified by the ASQ. To address the pre-vious report recommendation that the program track referrals for children identified as having a potential developmental de-lay, Healthy Families has revised its family update form. The form now contains questions, which supplement the ASQ, Program replaced substance abuse questionnaire. Program will track referrals for children with developmen-tal delays. Other Pertinent Information 34 OFFICE OF THE AUDITOR GENERAL and will be used to track whether children with developmental delays are offered appropriate referrals. The new form was im-plemented on July 1, 2000, and data from this measurement will be analyzed in the next evaluation. 35 OFFICE OF THE AUDITOR GENERAL STATUTORY ANNUAL EVALUATION COMPONENTS Pursuant to A.R.S. §41-1279.08, the Office of the Auditor General is required to include the following information in the annual program evaluation. C.1. Information on the number and characteristics of program participants. n Number of Participants—Since 1994, 4,421 families have participated in Healthy Families. Of these, 3,267 met program criteria for being officially engaged in the program. That is, they received at least four home visits from a home visitor. The remaining 1,154 re-ceived three or fewer home visits and were included as a comparison group when examining child abuse rates. n Mother’s Age, Marital Status, Income, Ethnicity, Education, Employment Status, and Risk Score— Information about the age, marital status, income, ethnicity, employment status, and educational status of participants is shown in Table 7 (see page 36). Be-cause in nearly all cases, the child’s mother is the pri-mary program participant, information is provided for mothers only and is presented separately for par-ticipant families and for comparison group families. Statistical analyses showed that these two groups were similar in most areas, with a few exceptions. Comparison group members were more likely than participants to be single and employed. On average, participants were also slightly older (average age of 22.9) than the comparison group (average age of 22.3). In addition, scores on the Family Stress Checklist risk assessment tool were slightly higher for participant families (average score of 38) than for comparison group families (average score of 37), and the differ-ence was statistically significant. No significant differ-ences were found when comparing annual household income, education level, or ethnicity. Statutory Annual Evaluation Components 36 OFFICE OF THE AUDITOR GENERAL Table 7 Healthy Families Program Participant and Comparison Group Member Profile January 1994 through May 2000 Program Participants N (total number of participants) =3,267 Comparison Group N (total number of comparison group members)=1,154 Mother’s age 11-15 6.0% 5.7% 16-19 34.6 37.8 20-25 34.2 36.1 26-30 13.1 11.4 31 and older 12.1 9.1 Marital status Single 68.2 76.7 Married 20.3 13.4 Divorced 3.1 3.3 Separated 3.5 3.6 Widowed .2 .3 Other/Unknown 4.6 2.8 Annual income $10,000 and under 61.9 62.5 $10,001-$15,000 20.4 19.2 $15,001-$20,000 9.0 8.7 $20,001-$30,000 5.8 7.9 $30,001 and higher 2.9 1.7 Mother’s ethnicity Hispanic 54.3 54.8 Caucasian 23.8 24.0 African-American 7.9 8.7 Native American 10.0 8.4 Asian-American .5 .2 Other 3.5 4.0 Percentage employed at intake 13.2 18.8 Full-time 6 2 . 9 6 9 . 1 Part-time 37.1 30.9 Average years of education 10.3 10.4 Average risk score on Family Stress Checklist (Range 5 to 85) 37.9 37.0 Source: Auditor General staff analysis of data provided by the Healthy Families Program. n Regions Served—The Healthy Families Program serves families in 10 of Arizona’s 15 counties. The number of participants served from each county is shown in Figure 2 (see page 37). Statutory Annual Evaluation Components 37 OFFICE OF THE AUDITOR GENERAL Figure 2 Healthy Families Program Number of Participants and Comparison Group Members by County January 1994 through May 2000 1,013 371 753 243 444 132 284 96 235 107 213 85 147 51 126 21 52 48 0 200 400 600 800 1,000 1,200 Number of Families Participants Comparison Group Members Source: Auditor General staff analysis of data provided by the Healthy Families Program. Maricopa Pima Cochise Coconino Yavapai Santa Cruz Yuma La Paz/ Pinal Mohave County n Number of Children, Household Size, Sex of Child, Health Insurance, and Public Assistance Usage—Table 8 (see page 38) presents information on program participants’ number of children, household size, and the proportion of female and male children served by the program. It also provides information on participants’ self-reported health insurance status and use of public assistance. Statistical analyses re-vealed that, on average, comparison group members had a slightly larger household size (average of 4.89) than participants (average of 4.65). No significant dif-ferences were found for the number of children in participant versus comparison group families or for the proportion of male versus female children served in each group. In addition, no group differences were found for health insurance status or for the proportion Statutory Annual Evaluation Components 38 OFFICE OF THE AUDITOR GENERAL of program participants using Temporary Assis-tance for Needy Families (TANF), food stamps, or Women, Infants and Children (WIC). Table 8 Healthy Families Program Participants and Comparison Group Members Profile of Family Characteristics January 1994 through May 2000 Program Participants Comparison Group N (total number of partici-pants) = 3,267 N (total number of comparison group members) = 1,154 Number of children 1 54.0% 55.2% 2-3 34.3 35.6 4-5 9.5 7.3 6 or more 2.1 1.8 Household size 2-3 33.0 30.6 4-5 39.2 37.4 6-7 19.4 21.1 8-9 6.5 7.2 10 or more 1.8 3.6 Sex of child Female 49.6 50.2 Male 50.4 49.8 Health insurance AHCCCS1 80.3 80.1 Private 9.3 9.1 Other 3.3 2.9 None 6.1 6.1 Unknown 1.0 1.2 Public assistance services used as reported at intake TANF2 24.7 23.3 Food Stamps 35.6 35.4 WIC3 80.1 78.9 1 Arizona Health Care Cost Containment System. 2 Temporary Assistance for Needy Families. 3 Women, Infants and Children Nutrition Program. Source: Auditor General staff analysis of data provided by the Healthy Families Program. Statutory Annual Evaluation Components 39 OFFICE OF THE AUDITOR GENERAL C.2. Information on contractors and program service providers. See Introduction and Background (page 6) for informa-tion on contractors and program service providers. C.3. Information on program revenues and expenditures. See Table 1 (page 7) for information on program revenues and expenditures. C.4. Information on the number and characteristics of enrollment and disenrollment and information from program participants on the reasons for each. As of May 31, 2000, there were 1,178 families enrolled in Healthy Families. Since 1994, 4,421 families have enrolled. Table 9 (see page 40) provides information about the length of participation for families who have exited the program. These families participated for a median of 195 days (or 6 and a-half months). Among families who were active participants as of May 31, 2000, the median dura-tion of participation was slightly more than 1 year (379 days). In addition, the January 1998 Healthy Families evaluation reported an attrition rate of 45 percent for fiscal year 1997. An attrition rate for a given fiscal year is computed by di-viding the number of families who exited the program during that year by the number who participated in the program at any point during the year. The corresponding attrition rates for fiscal years 1998, 1999, and 2000 were 42 percent, 40 percent, and 35 percent, showing a gradual improvement in retention. 1 Figure 3 (see page 40) shows the most common reasons participants exit the program. More than half terminate because they have moved or because the program could not contact the family after several attempts. Other rea-sons for termination include participants’ refusal of fur- 1 Thirty-five percent attrition rate for 2000 includes 11 months of data from fiscal year 2000 (July 1, 1999 through May 31, 2000). Statutory Annual Evaluation Components 40 OFFICE OF THE AUDITOR GENERAL ther services, refusal of a change in home visitor, and achievement of self-sufficiency. Table 9 Healthy Families Program Months of Participation for Families at Exit January 1994 through May 2000 Number Percentage 1-6 months 1,520 46.9% 7-12 months 773 23.9 13-18 months 382 11.8 19-24 months 197 6.1 25-30 months 128 3.9 31-36 months 90 2.8 37-48 months 97 3.0 49-60 months 53 1.6 Total 3,240 Source: Auditor General staff analysis of data provided by the Healthy Families Program. Figure 3 Healthy Families Program Reasons for Exit January 1994 through May 2000 n (total number of families who left the program) = 3,212 Source: Auditor General staff analysis of data provided by Healthy Families staff. Refused home visitor change 4% Self-sufficiency 9% Unable to contact 31% Refused services 14% Family moved 28% Other 14% Statutory Annual Evaluation Components 41 OFFICE OF THE AUDITOR GENERAL C.5. Information on the average cost for each participant in the program. See Table 1 (page 7) for information on the average cost for each program participant. C.6. Information concerning the progress of program participants in achieving goals and objectives. See Finding I (pages 13 through 19) for information on the program’s progress in reducing rates of child abuse and neglect. See Finding II (pages 21 through 26) for information re-garding the program’s progress in improving children’s health and participants’ home environment. Finding II includes information about immunization rates, devel-opmental screenings, child safety, and parent-child inter-action. See Section D (pages 42 through 43) for information on participants’ progress in increasing self-sufficiency through employment, and reducing their dependence on public assistance. C.7. Information on any long-term savings associated with program services. As discussed in Finding I (see pages 13 through 19), evaluators did not find statistically significant differences in the rates of substantiated child abuse reports for Healthy Families participants and the comparison group. Thus, evaluators cannot estimate potential savings asso-ciated with reduction in child abuse rates, examining such factors as costs of CPS investigations, foster care placements, and in-home services. C.8. Recommendations regarding program administra-tion. 1. To ensure compliance with its statutorily defined goal to reduce substance abuse, the Healthy Families Pro-gram should continue to implement new measures Statutory Annual Evaluation Components 42 OFFICE OF THE AUDITOR GENERAL for assessing participant substance abuse problems and tracking substance abuse referrals. 2. To ensure that families with children who receive scores in the delayed range on the ASQ receive ap-propriate referrals, the Healthy Families Program should continue to implement new measures for tracking these referrals. C.9. Recommendations regarding informational materials distributed through the program. Current and past evaluations of Healthy Families infor-mational materials show that materials distributed by the program are adequate to address program needs. Specifi-cally, Healthy Families provides program participants with information regarding medical care, mental health, employment, and education. Additionally, the program provides information to promote parent-child interaction and to assist parents with understanding their child’s de-velopment. D. Effect of the program on encouraging parental re-sponsibility in employment, self-sufficiency, and child safety. Document the income level and family size of those receiving program services. n Employment rates—As seen in Figure 4 (see page 44), employment rates for both mothers and fathers enrolled in Healthy Families generally increased over time. Healthy Families can influence these rates by providing referrals to job services. According to pro-gram data, 6.3 percent of mothers and 10.5 percent of fathers received a referral to job services within two months of enrollment, 10.2 percent of mothers and 12.8 percent of fathers received such a referral at 6 months, and 12.2 percent of mothers and 12.7 percent of fathers received a job-related referral at 12 months. Although employment rates show an increase, it is important to note that due to attrition, at each time period, there are fewer participants represented. Therefore, it is difficult to determine the extent to Statutory Annual Evaluation Components 43 OFFICE OF THE AUDITOR GENERAL which increases in employment rates are due to pro-gram participation or due to differences among peo-ple who remain in the program compared to those who drop out. n Public assistance—In order to conduct an analysis of public assistance usage, evaluators requested, data from DES,containing all records of Temporary Assis-tance to Needy Families (TANF) and food stamps re-ceived by Healthy Families participants and compari-son group members. However, after receiving the data, evaluators learned that information about sev-eral of the participants was missing. Therefore, it was determined that any analysis of public assistance us-age would provide inaccurate results. In addition, it was not possible to make additional data requests due to the substantial amount of time it would have taken to make the request, receive the data, test it for reli-ability, and analyze the results. n Child safety—See Finding II (pages 21 through 26) for information on child safety, including the results of the Child Safety Checklist assessment. n Income level and family size of those receiving program services—See C.1. (pages 35 through 38) of this section for income levels and family size of those receiving program services. Statutory Annual Evaluation Components 44 OFFICE OF THE AUDITOR GENERAL Figure 4 Healthy Families Program Employment Rates of Mothers and Fathers in Program January 1994 through May 2000 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Months in Healthy Families Program Mothers Fathers Source: Auditor General staff analysis of data provided by the Healthy Families Program. Intake 2 6 12 18 24 30 36 42 48 54 60 Percentage Employed OFFICE OF THE AUDITOR GENERAL Appendix OFFICE OF THE AUDITOR GENERAL (This Page Intentionally Left Blank) a-i OFFICE OF THE AUDITOR GENERAL Appendix Assessment Tools n Ages and Stages Questionnaire (ASQ)—The ASQ is a developmental screening tool that is completed by the parent and is used to assess whether children are developing nor-mally, both physically and socially. The questionnaire ad-dresses five areas of child development: (1) Communication, (2) Gross Motor Skills, (3) Fine Motor Skills, (4) Problem Solv-ing, and (5) Personal Social Skills. For each area, parents are asked to respond to six questions about whether their chil-dren are engaging in behavior appropriate for their age. The ASQ is administered at the following ages: 4, 6, 12, 18, 24, 30, 36, and 48 months. Reference: Bricker, Diane, Jane Squires, Linda Mounts, La- Wanda Potter, Bob Nickel, and Jane Farrell. The Ages and Stages Questionnaire: A Parent-Completed, Child-Monitoring Sys-tem. Baltimore: Paul H. Brookes Publishing Co., 1995. n The CAGE Questionnaire—Substance Abuse Screen-ing— The CAGE Questionnaire was designed to identify po-tential alcohol abuse problems. It was also modified to in-clude the abuse of drugs other than alcohol. The acronym “CAGE” stands for the first letter in each of four questions that are asked of respondents: (1) Have you ever felt the need to cut down on drinking/drug use? (2) Have you ever felt annoyed by others’ criticism of your drinking/drug use? (3) Have you ever felt guilty about your drinking/drug use? and (4) Have you ever had a drink/taken drugs first thing in the morning (eye-opener)? Two or more “yes” responses are considered indicative of a substance abuse problem. The CAGE is administered after 6 and 12 months of program par-ticipation. Appendix a-ii OFFICE OF THE AUDITOR GENERAL Reference: Mayfield, D., G. McCleod, and P. Hall. The CAGE Questionnaire: Validation of a New Alcoholism Screening Questionnaire. American Journal of Psychiatry, 1974, 131, 1121- 1123. n The Child Safety Checklist (CSC)—The Child Safety Checklist is an instrument that assesses whether various safety measures in the home have been implemented. The CSC is administered by home visitors who ask parents whether or not each safety measure on the checklist has been taken (for example, “do you use a car seat for your baby?”). There are two versions of the child safety checklist. The first is administered when the child is 2, 6, 12, 18, 24, and 30 months of age. The second contains questions designed for families with older children and is administered at the fol-lowing ages: 36, 42, 48, 54, and 60 months. n Parenting Stress Index (PSI)—The PSI is an instrument designed to identify stressful situations that could potentially put parents at risk for “dysfunctional parental behavior,” in-cluding abuse. It includes several subscales that measure stress related to child characteristics and parent functioning. For the evaluation, seven subscales were used. These in-cluded two that focused on child characteristics (child’s mood and distractibility/hyperactivity) and five that fo-cused on adult characteristics (depression, attachment, re-striction of role, sense of competence, and social isolation). The PSI is administered after 3 weeks, than at 6 and 12 months of program participation. Reference: Abidin, Richard R. Parenting Stress Index Admini-stration Manual, Third Edition. Odessa, FL: Psychological As-sessment Resources, Inc., 1995. OFFICE OF THE AUDITOR GENERAL Agency Response OFFICE OF THE AUDITOR GENERAL (This Page Intentionally Left Blank) ARIZONA DEPARTMENT OF ECONOMIC SECURITY 1717 W. Jefferson - P.O. Box 6123 - Phoenix, AZ 85005 Jane Dee Hull John L. Clayton Governor Director Ms. Debbie Davenport, CPA Auditor General 2910 North 44th Street, Suite 410 Phoenix, AZ 85018 Dear Ms. Davenport: The Department wishes to thank the Office of the Auditor General for the opportunity to respond to the recently completed audit of the Healthy Families Arizona Program. Report findings II and III indicate that the program benefits the families we serve. Finding I is inconclusive and is based on an indicator (child abuse report rates) that is a relatively low occurring event in the participant families, the comparison group, and in the general population. It is important that we keep in mind that the reason the program exists and the reason we focus resources on prevention is to help ensure that no child is maltreated and that all children have the chance to be nurtured and comforted during their growing years into adulthood. Because of the years of positive evaluation outcomes, we believe the Healthy Families Arizona Program is an effective prevention program. The Department welcomes the opportunity to work with you in helping to find other methods of determining the impact of the program’s positive outcomes on the prevention of child abuse and neglect. The Department will continue to utilize the new forms listed in the two recommendations under Finding II. Finally, please accept our appreciation for the time and effort invested in this important evaluation. We wish to specifically recognize Laurie Cohen and JoAnne Dukeshire for their dedication during the evaluation process. Sincerely, John L. Clayton DEPARTMENT OF ECONOMIC SECURITY RESPONSE TO THE HEALTHY FAMILIES ARIZONA PROGRAM EVALUATION FINDING I: Healthy Families Participant Child Abuse Rates Do Not Differ Significantly from the Comparison Group This report has found no significant difference between Healthy Families participants and the comparison group on the rates of substantiated CPS reports. The report states that there are three possible explanations for these results. The true reason may never be known as, but we do realize that using substantiated child abuse and neglect reports as a measure for program success is fraught with difficulties. State child abuse data often are considered to be the primary indicator of successful prevention efforts. However, if one wishes to obtain an accurate appraisal of family functioning, using CPS data may not be adequate. For example, a national study found that one third of child maltreatment incidents go unreported (Sedlak & Broadhurst, 1996), so reliance on this data will produce an incomplete picture. Observations of interactions between a parent and child, maternal warmth, sensitivity, and nurturing are better indicators of parent-child relationships than maltreatment reports. Healthy Families Arizona independent evaluations have time and again over the years showed excellent outcomes in these meaningful areas. The Department will cooperate in any way possible in helping to find or develop a better method of determining the impact of the program’s positive outcomes on the prevention of child abuse and neglect. There are no recommendations under this finding. DEPARTMENT OF ECONOMIC SECURITY RESPONSE TO THE HEALTHY FAMILIES ARIZONA PROGRAM EVALUATION FINDING II: Healthy Families Program Provides Health Referrals, Encourages On-Time Immunizations and Promotes Safety The Department agrees the Healthy Families Arizona Program provides health referrals, encourages on-time immunizations and promotes safety in the program’s participant families. The Department is proud of the many successes in these areas. An example of one success is how the direct involvement of a Healthy Families home visitor saved the life of a family of nine. An early morning fire swept through the two-bedroom apartment of a participant family. The mother was alerted in time because the Healthy Families home visitor had recently completed the Child Safety Checklist with the family and had helped them purchase and install smoke detectors in the apartment. The finding of the Auditor General is agreed to and the audit recommendation will be implemented which states that to ensure compliance with its statutorily defined goal to reduce substance abuse, the Healthy Families Program should continue to implement the new questionnaire for assessing participant substance abuse problems and method for tracking substance abuse referrals. The Healthy Families Program started using the new substance abuse assessment questionnaire on July 1, 2000 to measure the effect of the program on reducing dependency on alcohol and drugs over time. Questions have been added to the family update form to enable staff to track referrals for substance abuse treatment. The finding of the Auditor General is agreed to and the audit recommendation will be implemented which states that to ensure families with children who receive scores in the delayed range on the Ages & Stages Questionnaire (ASQ) receive appropriate referrals, the Healthy Families Program should continue to implement new measures for tracking these referrals. As this report states, the Healthy Families Program already revised the family update form, which was implemented on July 1, 2000. The form includes tracking reporting information on referrals for children identified as having a potential developmental delay. This new form contains questions which supplement the ASQ, and is being used to track whether children with developmental delays are offered appropriate referrals. DEPARTMENT OF ECONOMIC SECURITY RESPONSE TO THE HEALTHY FAMILIES ARIZONA PROGRAM EVALUATION FINDING III: Parenting Stress Measure Shows Improved Family Relations The Department agrees the Healthy Families Program has improved family relations in participant families. The outcomes from the Parenting Stress Index show that parental stress decreases after 12 months of participation in the program. As pointed out in this report, research shows that high scores on parenting stress are related to the potential to abuse one’s child, and, conversely, low to normal scores are related to positive parent-child relationships. The program also assists participants with higher stress scores to seek mental health services, thus providing help to parents so that they have a better opportunity to become nurturing parents to their children. The Department fully intends to continue independent evaluation of the Healthy Families Arizona Program and to maintain national accreditation to ensure continued excellence in program delivery and administration. There are no recommendations under this finding. Other Performance Audit Reports Issued Within the Last 12 Months 00-4 Family Builders Pilot Program 00-5 Arizona Department of Agriculture— Licensing Functions 00-6 Board of Medical Student Loans 00-7 Department of Public Safety— Aviation Section 00-8 Arizona Department of Agriculture— Animal Disease, Ownership and Welfare Protection Program 00-9 Arizona Naturopathic Physicians Board of Medical Examiners 00-10 Arizona Department of Agriculture— Food Safety and Quality Assurance Program and Non-Food Product Quality Assurance Program 00-11 Arizona Office of Tourism 00-12 Department of Public Safety— Scientific Analysis Bureau 00-13 Arizona Department of Agriculture Pest Exclusion and Management Program 00-14 Arizona Department of Agriculture State Agricultural Laboratory 00-15 Arizona Department of Agriculture— Commodity Development Program 00-16 Arizona Department of Agriculture— Pesticide Compliance and Worker Safety Program 00-17 Arizona Department of Agriculture— Sunset Factors 00-18 Arizona State Boxing Commission 00-19 Department of Economic Security— Division of Developmental Disabilities 00-20 Department of Corrections— Security Operations 00-21 Universities—Funding Study 00-22 Annual Evaluation—Arizona’s Family Literacy Program 01-01 Department of Economic Security— Child Support Enforcement Future Performance Audit Reports Department of Public Safety—Telecommunications Department of Public Safety—Drug Abuse Resistance Education (D.A.R.E.) Program Board of Osteopathic Examiners in Medicine and Surgery |