Performance audit, Arizona Supreme Court: Administrative Office of the Courts - juvenile treatment programs : a report to the Arizona Legislature

              A REPORT

ARIZONA LEGISLATURE

TO THE



Performance Audit Division Performance Audit



Arizona Supreme Court

Administrative Office of the Courts-- Juvenile Treatment Programs



December � 2007 REPORT NO. 07-13



Debra K. Davenport Auditor General



The Auditor General is appointed by the Joint Legislative Audit Committee, a bipartisan committee composed of five senators and five representatives. Her mission is to provide independent and impartial information and specific recommendations to improve the operations of state and local government entities. To this end, she provides financial audits and accounting services to the State and political subdivisions, investigates possible misuse of public monies, and conducts performance audits of school districts, state agencies, and the programs they administer.



The Joint Legislative Audit Committee

Senator Robert Blendu, Chair Senator Senator Senator Senator Senator Carolyn Allen Pamela Gorman Richard Miranda Rebecca Rios Tim Bee (ex-officio) Representative John Nelson, Vice-Chair Representative Representative Representative Representative Representative Tom Boone Jack Brown Peter Rios Steve Yarbrough Jim Weiers (ex-officio)



Audit Staff

Melanie M. Chesney, Director Dale Chapman, Manager and Contact Person Sara Bessette Robin Hakes Brian Miele Jeremy Weber



Copies of the Auditor General's reports are free. You may request them by contacting us at:



Office of the Auditor General

2910 N. 44th Street, Suite 410 � Phoenix, AZ 85018 � (602) 553-0333

Additionally, many of our reports can be found in electronic format at:



www.azauditor.gov



STATE OF ARIZONA

DEBRA K. DAVENPORT, CPA AUDITOR GENERAL



OFFICE OF THE



AUDITOR GENERAL



WILLIAM THOMSON DEPUTY AUDITOR GENERAL



December 3, 2007 Members of the Arizona Legislature Mr. David K. Byers, Director Administrative Office of the Courts Ms. Carol Boone, Chief Juvenile Probation Officer Maricopa County Juvenile Probation Department The Honorable Janet Napolitano, Governor Mr. Rik Schmidt, Director Juvenile Court Services Pima County Juvenile Court



Transmitted herewith is a report of the Auditor General, a Performance Audit of the Arizona Supreme Court, Administrative Office of the Courts (AOC)--Juvenile Treatment Programs. This report is in response to Arizona Revised Statutes (A.R.S.) �41-2958 and was conducted under the authority vested in the Auditor General by A.R.S. �41-1279.03. I am also transmitting with this report a copy of the Report Highlights for this audit to provide a quick summary for your convenience. As outlined in its response, the AOC agrees with the findings and plans to implement the recommendations. In addition, responses from the Maricopa County Juvenile Probation Department and the Pima County Juvenile Court are included. My staff and I will be pleased to discuss or clarify items in the report. This report will be released to the public on December 4, 2007. Sincerely, Debbie Davenport Auditor General Enclosure cc: The Honorable Ruth V. McGregor Chief Justice of the Arizona Supreme Court The Honorable Eileen Willett Presiding Juvenile Court Judge Maricopa County Juvenile Court The Honorable Patricia Escher Presiding Juvenile Court Judge Pima County Juvenile Court



2910 NORTH 44



th



STREET � SUITE 410 � PHOENIX, ARIZONA



85018 � (602) 553-0333 � FAX (602) 553-0051



SUMMARY

The Office of the Auditor General has conducted a performance audit of the Arizona Supreme Court, Administrative Office of the Courts (AOC)--Juvenile Treatment Programs pursuant to Arizona Revised Statutes (A.R.S.) �41-2958, which requires a review of the programs and commissions established by the Legislature within the judiciary. This audit was conducted under the authority vested in the Auditor General by A.R.S. �41-1279.03. The Supreme Court (Court), in coordination with county juvenile courts, administers juvenile supervision and treatment programs throughout the State. In fiscal year 2006, almost 48,400 juveniles were referred to juvenile courts. Through a process called diversion, many juveniles avoid formal court action by accepting consequences specified by court probation officers or citizen boards. Other juveniles may be formally charged and ordered to pay a monetary penalty and/or complete unpaid community services, be placed on probation, or be committed to the Arizona Department of Juvenile Corrections. Juveniles in the court system may be required to participate in treatment services or other programs designed to reduce the risk of re-offense. These services and programs may include drug testing and substance abuse programs, residential treatment programs, family therapy, counseling, professional evaluations, and education services. Funding for these services comes mainly from state funding, Medicaid, family contributions, and/or private insurance. The Court, through the AOC, is responsible for managing state monies used for these services, which totaled nearly $22.8 million in fiscal year 2007.



Supreme Court should improve assessment, planning, and monitoring processes (see pages 11 through 24)

Under the direction of the Arizona Judicial Council, the AOC should work with the county juvenile courts to improve the processes that counties use to identify, plan for, and monitor the provision and effectiveness of treatment services to juveniles. Auditors identified opportunities for improvement in the following five areas:



The Arizona Judicial Council assists the Supreme Court and the chief justice in developing and implementing policies and procedures for the administration of all courts. For example, it studies the internal operation of the courts and plans for future developments. It also promotes improvements and responds to issues concerning judicial administration by reviewing and recommending for adoption by the chief justice proposed administrative orders, code sections, rules, and policies.



Office of the Auditor General

page



i



R Risk assessments--Although the Court requires counties to complete a courtdeveloped risk assessment for all juveniles who are referred to county juvenile courts, some juveniles do not receive such assessments. Data from the Juvenile Online Tracking System (JOLTS) showed that for the 12,591 juveniles on probation in fiscal year 2006, 95 percent had at least one completed risk assessment. However, juveniles should receive a risk assessment each time they are referred to court and more than 67 percent of the 12,591 juveniles had at least one court referral for which no risk assessment was conducted before disposition (outcome for a juvenile), and 25 percent had at least one court referral for which no risk assessment was completed at all. Without completed risk assessments, courts lack important information to determine the appropriate level of supervision for juveniles on probation and treatment services for a juvenile. Auditors identified four ways that the AOC could work to improve counties' use of the risk-assessment instrument--providing guidelines on the time frame in which a risk assessment must be completed; providing guidance on when, and under what circumstances, one risk assessment may be performed to address multiple referrals; developing policies for using a juvenile's risk assessment to help determine the appropriate level of supervision for juveniles on probation and treatment services; and using the risk assessment to determine which juveniles should receive further court assessments. N Needs assessments--Counties must also complete the Court's needs assessment for all juveniles who have been adjudicated, but many juveniles do not receive these assessments. JOLTS data showed that nearly 40 percent of the 12,591 juveniles on probation in fiscal year 2006 had at least one instance for which no court-developed needs assessment was conducted prior to disposition and nearly 46 percent had at least one instance for which the Court's needs assessment was not completed at all. Auditors' review of a random sample of 25 juvenile case files in Maricopa and Pima Counties found that these counties were using other needs assessment tools and methods. The AOC is working with representatives from 11 county juvenile courts, Arizona State University, and the Arizona Department of Juvenile Corrections to determine if its needs assessment tool conforms to best practices from other states and agencies, or if assessment tools from another state or agency can better meet its needs. S Strengths/protective factors assessments--Although the Court has developed a strengths/protective factors assessment, it does not require its use. The Court's strengths/protective factors assessment identifies positive influences and traits that may prevent a juvenile from further involvement with the courts. However, auditors' review of files for the 25 juveniles on probation in the two counties showed that 23 did not receive the Court's strengths/protective factors assessment. According to the National Council of Juvenile and Family Court Judges, it is important to use a formal structured assessment of both needs and



State of Arizona

page



ii



strengths as the foundation for the case plan.1 Therefore, the AOC should evaluate and, if necessary, revise its strengths/protective factors assessment and establish policy requiring this assessment's completion. C Case plans--The AOC should also take steps to improve the case plans developed for juveniles involved in the juvenile justice system. The National Center for Juvenile Justice's Desktop Guide recommends developing such plans with specific goals, measurable behavioral objectives, action steps, and completion dates.2 Although the Court requires probation officers to develop case plans, it does not have policies or procedures that guide how these plans should be developed or used. Consequently, practices vary in the two counties that auditors reviewed. For the 15 juvenile case files that auditors reviewed from Maricopa County, plan development was sporadic; no formal plan was present for nearly half of the instances where these juveniles were placed on probation. In Pima County, although each of the 10 cases reviewed had a section in the disposition report that provided information on needs and treatment service recommendations, some of these sections lacked specific goals, action steps, or time frames for accomplishing goals or completing services. Therefore, the AOC should develop policies and procedures regarding the development of case plans and the information these plans should contain. C Case monitoring--The National Council of Juvenile and Family Court Judges' Juvenile Delinquency Guidelines calls for active monitoring of cases, and the states that auditors contacted require reassessments of a juvenile's risk and needs at regular intervals.3 In Arizona, however, probation officers inconsistently monitor juveniles' progress and do not consistently reassess their needs based on progress made or update case plans. As a result, juveniles may not receive the services they need. Auditors' review of the sample of the case files for 25 Maricopa and Pima County juveniles on probation found that 5 of these juveniles had not received treatment services as ordered by the juvenile court, services that the Court typically funds. Additionally, AOC officials indicated that the AOC does not centrally monitor services received through funding sources other than the Supreme Court, and according to probation supervisors, neither Maricopa nor Pima County has policies and procedures for conducting reassessments to measure juveniles' progress and/or reassess risk levels. The AOC should develop and implement policies and procedures for monitoring juveniles' progress and updating case plans accordingly.



1



Wiebush, Richard G. Graduated Sanctions for Juvenile Offenders: A Program Model and Planning Guide. Reno, NV: National Council of Juvenile and Family Court Judges, Juvenile Sanctions Center, 2003. National Center for Juvenile Justice. Desktop Guide to Good Juvenile Probation Practice. Pittsburgh, PA: National Center for Juvenile Justice, 2002. Grossmann, David E. and Maurice Portley. Juvenile Delinquency Guidelines: Improving Court Practice in Juvenile Delinquency Cases. Reno, NV: National Council of Juvenile and Family Court Judges, 2005.



2



3



Office of the Auditor General

page



iii



Contracting process thorough, but AOC should improve monitoring of treatment services vendors (see pages 25 through 32)

The Supreme Court, through the AOC, has established a thorough process to contract with treatment services vendors, but it should take some steps to improve its monitoring of these vendors. The Court's comprehensive contracting process includes such steps as creating a list of qualified vendors, seeking input from counties on services needed, soliciting and evaluating proposals, and awarding contracts. The Court also awards the contracts in such a way as to allow for changing needs or circumstances during the contract period--typically 5 years. Although the contracting process is comprehensive, the AOC should improve its monitoring of its contracted vendors. Specifically: R Reviewing small vendors--Limited staff resources prevent the AOC from conducting site visits of all vendors. The AOC focuses its site visits on vendors that have submitted billings in a particular service category of more than $20,000 during the previous calendar year. Although vendors that may be providing many types of services, each costing less than this amount, are generally not included in the monitoring site visits, according to AOC officials, it may include vendors that fall slightly below that figure. According to a 2001 review of this function, AOC internal auditors recommended that the AOC consider periodically monitoring smaller vendors, as it may be beneficial to independently validate compliance with certain contract provisions, such as fingerprinting employees and documenting service provision. Therefore, the AOC should modify its approach for selecting vendors for site visits to conduct annual site visits for a sample of its smaller vendors. E Ensuring timely correction of deficiencies--The AOC sometimes takes months to respond to a vendor's corrective action plan to resolve all of the issues in the monitoring report. The AOC should finalize, implement, and follow its policies for the timely review and approval of these plans.



Other Pertinent Information (see pages 33 through 36)

Auditors also collected information regarding the Interagency Integration Coordination Initiative, a state-wide effort begun in 2006 to coordinate treatment services between multiple state agencies for dependent and delinquent juveniles and their families. Participants have worked to promote greater integration of the services provided to children and families by state systems, including behavioral



State of Arizona

page



iv



health, juvenile justice, and child welfare systems. Participants also identified four potential outcomes of the initiative, as well as the associated strategies and action steps to reach these outcomes. The potential outcomes range from sharing data to reducing cases of delinquent and dependent juveniles within the child welfare and juvenile justice systems.



Office of the Auditor General

page



v



State of Arizona

page



vi



TABLE OF CONTENTS

Introduction & Background Finding 1: Supreme Court should improve assessment,

planning, and monitoring processes

Supreme Court's risk- and needs- assessment tool Risk assessment not fully used Needs and strengths/protective factors assessments not fully used Case planning needs improvement AOC and counties should improve treatment services monitoring Recommendations



1



11 11 13 15 17 20 22



Finding 2: Contracting process thorough, but AOC should

improve monitoring of treatment services vendors

AOC administers and manages treatment services contracts Supreme Court has comprehensive contracting process AOC should strengthen vendor monitoring Recommendations



25 25 26 28 32 33 37



Other Pertinent Information Bibliography



continued



Office of the Auditor General

page



vii



TABLE OF CONTENTS

Tables:

1 Expenditures for Supreme Court-Funded Juvenile Treatment Services Fiscal Year 2007 (Unaudited) Expenditures for Supreme CourtFunded Juvenile Treatment Services By County Fiscal Year 2007 (Unaudited)



2



2



26



concluded



State of Arizona

page



viii



INTRODUCTION & BACKGROUND

The Office of the Auditor General has conducted a performance audit of the Arizona Supreme Courts--Administrative Office of the Courts (AOC)--Juvenile Treatment Programs pursuant to Arizona Revised Statutes (A.R.S.) �41-2958, which requires a review of the programs and commissions established by the Legislature within the judiciary. This audit was conducted under the authority vested in the Auditor General by A.R.S. �41-1279.03.



Supreme Court administers juvenile supervision and treatment programs

The Supreme Court (Court), in coordination with county juvenile Delinquent juvenile--A juvenile who courts, administers juvenile supervision and treatment programs commits an illegal offense that would be to protect communities and rehabilitate delinquent and considered criminal if committed by an incorrigible juveniles. Juveniles may be referred to the courts for adult. delinquent or incorrigible offenses by parents, school officials, Incorrigible juvenile--A juvenile who police officers, probation officers, or others. Referred juveniles commits an offense that would not be may be placed on diversion or proceed to formal court hearings. considered criminal if committed by an Diverted juveniles avoid formal court action by accepting adult, such as habitual truancy from school, consequences specified by court probation officers or citizen violating curfew, or refusal to obey boards established by county attorneys and/or the juvenile reasonable and proper directions from courts.1 If a juvenile proceeds to formal court hearings, the courts parents or guardians. conduct adjudication and disposition hearings for the juvenile, Source: The Supreme Court's Juvenile Justice Services Division similar to court trials and sentencing hearings in the adult court Web site. system, but without a jury. The hearings can result in juveniles being ordered to pay a monetary penalty and/or complete unpaid community service, placed on probation, or committed to the Arizona Department of Juvenile Corrections. The courts may also require juveniles to participate in treatment services or other programs designed to reduce the risk of re-offense. These services and programs may include drug testing and substance abuse programs, residential treatment programs, family therapy, counseling, professional evaluations, and education services.



Delinquent and Incorrigible Juveniles



1



To encourage community participation in the juvenile justice system, county attorneys and/or the juvenile courts have created community-based alternative programs in which panels of citizens hear diverted juveniles' cases and assign consequences. Office of the Auditor General

page



1



Statute requires the juvenile courts to use a common risk/needs assessment for all referred juveniles.



The courts use various assessments to evaluate juveniles' risk of re-offending and to determine the most appropriate level of supervision for a juvenile. Specifically, A.R.S. �8-246(C) requires the juvenile courts to use a common risk/needs assessment for all referred juveniles to determine appropriate dispositions (sanctions, probation, or detention), which may include treatment services. In response, the Supreme Court developed risk and needs assessments and requires county juvenile courts to complete risk assessments for all referrals and needs assessments for all adjudicated referrals. In addition to the Court's assessments, juvenile courts in some counties use other assessments to evaluate juveniles' needs. Assessments are completed by probation officers, who also recommend dispositions and treatment services to the courts. Juvenile court judges can then use this information to determine a juvenile's disposition, which could include ordering a juvenile's participation in specific treatment services.



Table 1:



When a juvenile is ordered to participate in treatment services, the juvenile's assigned probation officer is typically responsible for communicating and coordinating with service providers and developing resources and opportunities for treatment and rehabilitation. The Expenditures for Supreme Court-Funded probation officer is also responsible for monitoring Juvenile Treatment Services treatment services provided to juveniles under their Fiscal Year 2007 supervision. (Unaudited)

Amount $ 7,317,276 6,518,756 2,388,607 1,250,916 1,214,326 1,170,854 1,069,326 778,270 565,659 177,104 142,871 102,690 46,165 17,240 9,245 3,038 563 $22,772,906



Treatment Services Category Out-of-home-care Sex offender services Substance abuse services Delinquency prevention/intervention services Outpatient mental health Evaluation and diagnosis Ancillary services (drug testing, polygraphs, etc.) Multisystemic therapy services Behavioral support services Renewing Arizona family traditions (intensive, inhome therapeutic interventions) Functional family therapy Competency restoration (education programs) Foster home services Substance abuse intensive outpatient program Brief strategic family therapy Education services (tutoring and GED testing) Violence intervention/prevention services Total

Source:



Auditor General staff analysis of fiscal year 2007 expenditure information from the Supreme Court's juvenile treatment services invoice-tracking system; Juvenile Treatment Services Fund Invoice Billing Manual for contract year 2007; and the Renewing Arizona Family Traditions 2006 annual report.



The Supreme Court, through the AOC, administers state funding and contracts with vendors for treatment services for court-involved juveniles. In fiscal year 2007, the AOC contracted with 161 treatment service providers for 17 categories of treatment services. These services included out-ofhome care, sex offender and substance abuse services, therapy, and evaluations. Treatment services are primarily funded through state funding, Medicaid, family contributions, and/or private insurance. According to A.R.S. �8-243(A), juveniles and their parents are responsible for any portion of the treatment costs that the juvenile courts determine they can pay. In addition, the AOC requires the juvenile courts to check each juvenile's enrollment status in Arizona's Medicaid program. In fiscal year 2007, the Supreme Court paid nearly $22.8 million for treatment services billed by 151 providers. Table 1 shows the fiscal year 2007 expenditures for courtfunded treatment services by treatment service category.



State of Arizona

page



2



According to AOC staff analysis, almost 48,400 juveniles were referred to Arizona's juvenile courts in fiscal year 2006. According to Auditor General staff analysis of court data, 12,591 juveniles were on some form of probation at some point during fiscal year 2006.1 Of the juveniles on probation, 10,865 had received state-funded services while in the juvenile court system. These juveniles received a median value of $643 in state-funded treatment services, ranging from $6 to more than $222,000 per juvenile.



Nearly 48,400 juveniles were referred to Arizona's juvenile courts in fiscal year 2006.



Court initiatives for improving juvenile treatment services and court processes

Beginning in fiscal year 2007, the Supreme Court, through the AOC, has undertaken two initiatives for improving juvenile treatment services and court processes. These initiatives are the implementation of a treatment program evaluation tool and promoting implementation of juvenile delinquency guidelines by the county juvenile courts. Specifically: S Standardized Program Evaluation Protocol (SPEP)--Between August 2006 and May 2007, the AOC implemented the SPEP to evaluate and improve the effectiveness of juvenile treatment programs contracted by the AOC.2 The SPEP is a method for evaluating juvenile treatment programs against current researchbased best practices. It scores programs based on how their characteristics compare with those that research has shown to reduce recidivism. The SPEP also identifies specific options for improving the evaluated programs. In addition to program evaluation, the AOC believes that the SPEP in conjunction with the , risk and needs assessment tools, will help the juvenile courts effectively refer juveniles to appropriate treatment programs. In May 2007, the AOC distributed SPEP scores for the programs evaluated during initial implementation. According to AOC management, 59 programs from 44 of the 161 service providers under contract in fiscal year 2007 were evaluated and account for an estimated 70 percent of dollars spent on treatment. To improve these programs, the AOC required that service providers submit program improvement plans for all programs that received a SPEP score by July 2007 and implement these plans by November 2007. Although not all providers complied with the July deadline, AOC management stated that they are working to engage providers through collaboration and technical assistance to encourage compliance. For example, AOC management said that they held workshops to assist providers in developing plans. AOC management also said they made the providers' SPEP scores and program improvement plans

1 2 Fiscal year 2007 data on juveniles in the juvenile court system was not available at the time of the data request. In implementing the SPEP the AOC collaborated with a group of national experts directed by Dr. Mark Lipsey, Director of , the Center for Evaluation and Research Methodology at the Vanderbilt Institute for Public Policy Studies, and Dr. James Howell, former Director of Research and Program Development at the federal Office of Juvenile Justice. Office of the Auditor General

page



3



available to the juvenile courts in September 2007 through the Court's intranet and continue to provide SPEP training for juvenile court personnel. According to management, the AOC plans to evaluate programs that were not initially evaluated by July 2008 and is working with Vanderbilt University to determine an appropriate frequency for evaluating and re-evaluating vendors' programs. J Juvenile Delinquency Guidelines (Guidelines)--The AOC has been working with the county juvenile courts to encourage implementation of the Guidelines developed by the National Council of Juvenile and Family Court Judges (National Council).1 Published in 2005, the Guidelines identify 16 key principles that promote excellence in juvenile delinquency proceedings. In addition, the Guidelines identify preferred practices for the entire court process that are designed to improve delinquency case processing and outcomes. For example, the Guidelines state that courts should have sufficient and accessible services in order to be reasonably assured that they can meet the needs of delinquent juveniles, which include holding juveniles accountable and assisting juveniles to learn new attitudes and competencies that result in law-abiding behavior. Courts formally implement the guidelines through contracts with the National Council. According to the National Council, contracts call for a multi-year relationship with the National Council, which provides assessment, planning, technical assistance, and evaluation services. Courts that contractually commit to implementing the Guidelines are designated as Model Delinquency Courts by the National Council. Model Delinquency Courts are responsible for securing their own funding for implementation. In December 2005, the Pima County juvenile court became the first Model Delinquency Court in the nation. According to a Pima County juvenile court official, Pima County is funding the implementation with a United States Department of Justice, Bureau of Justice Assistance grant. Between May and August 2007, the AOC conducted three regional trainings with 14 county juvenile courts to assess county interest in implementing the Guidelines. At the regional trainings, the courts completed self-assessments and developed action plans for specific guidelines on which they chose to focus. The county courts (except for Pima) are implementing the Guidelines through Arizona's Model Delinquency Guidelines Initiative funded through the Supreme Court. In September 2007, AOC management indicated that the Supreme Court was developing a contract with the National Council to provide technical assistance to ten Arizona counties to help implement their action plans, and has budgeted $80,000 from Juvenile Crime Reduction monies for this initiative (see page 6 for information about these monies).



1



Grossmann, David E. and Maurice Portley. Juvenile Delinquency Guidelines: Improving Court Practice in Juvenile Delinquency Cases. Reno, NV: National Council of Juvenile and Family Court Judges, 2005.



State of Arizona

page



4



State funding for juvenile supervision and treatment programs

In fiscal year 2007, the State appropriated more than $54 million for juvenile supervision and treatment programs. The AOC allocates state funding to the juvenile courts through several programs and funds created by state law for the supervision and treatment of juveniles. These programs and funds include the following: J Juvenile Probation Services Fund (JPSF)--Established by A.R.S. �8-322, the JPSF funds treatment services for juveniles on probation. According to AOC staff, some JPSF monies are also used to pay for superior court personnel and other expenses. In addition, AOC staff indicated that the Supreme Court retains a portion of the fund for administrative costs and state-wide projects that indirectly support probation services. Although the JPSF largely consists of State General Fund monies, it also includes monies reimbursed by juveniles' parents to provide treatment services. State law requires the juvenile courts to attempt to collect and then remit these monies to the Supreme Court, which then uses the monies to offset county allocations of state funds. In fiscal year 2007, the Legislature appropriated approximately $22.5 million in State General Fund monies to the JPSF. According to AOC staff, approximately $391,000 in parental payments was remitted to the Court in fiscal year 2007. D Diversion--A.R.S. �8-322 authorizes the Supreme Court to fund programs for diverted juveniles. According to AOC management, diversion funding is used for court personnel expenses and treatment services associated with county diversion programs. In fiscal year 2007, the Legislature appropriated approximately $10.2 million in State General Fund monies for diversion. J Juvenile Standard Probation Program--Under this program, delinquent or incorrigible juveniles are placed under the Court's care and control and are supervised by probation officers to ensure that they comply with court-ordered terms, including participation in treatment services. By law, a probation officer may supervise no more than an average of 35 juveniles on probation at one time. Under A.R.S. �12-261 et seq., state funding is allocated to county juvenile courts for improving, maintaining, or expanding juvenile probation services. State funding is primarily used to pay probation officer salaries and to help maintain the 35:1 ratio of juveniles to probation officers. In fiscal year 2007, the Legislature appropriated approximately $5.2 million in State General Fund monies to this Program. J Juvenile Intensive Probation Supervision (JIPS) Program--JIPS is a more structured and supervised probation program for juveniles who have committed serious or numerous offenses. Established by A.R.S. �8-351 et seq., JIPS was



Office of the Auditor General

page



5



created to reduce the commitment of juveniles to the Arizona Department of Juvenile Corrections and other residential treatment facilities. Under JIPS, juveniles are supervised by probation teams consisting of one to three probation and surveillance officers. Two-person and three-person probation teams may supervise no more than 25 and 40 JIPS probationers, respectively. In smaller counties, a single probation officer may supervise no more than 15 JIPS probationers. JIPS monies pay for court personnel, program operating costs, and treatment services. In fiscal year 2007, the Legislature appropriated approximately $10.4 million in State General Fund monies to this Program. J Juvenile Crime Reduction--Juvenile crime reduction monies are used for the development and initial implementation of programs designed to prevent juvenile crime and provide intervention services. These monies are appropriated from the Criminal Justice Enhancement Fund. The Criminal Justice Enhancement Fund was established by A.R.S. �41-2401 for the purpose of enhancing county jail facilities and operations, and is composed of surcharges on all court fines. By law, 9.35 percent of the Criminal Justice Enhancement Fund is dedicated for reducing juvenile crime. In fiscal year 2007, the Legislature appropriated approximately $5.2 million from the Criminal Justice Enhancement Fund to juvenile crime reduction. F Family Counseling Program--Family Counseling Programs are established by rules and guidelines promulgated by the presiding juvenile court judge in each county to strengthen family relationships and prevent juvenile delinquency. Program monies are used to provide counseling services for delinquent and incorrigible juveniles and their families. This Program requires the counties to match $1 for every $4 from the State. In fiscal year 2007, the Legislature appropriated $660,400 in State General Fund monies to this program. D Drug Court Program--Drug courts are coordinated programs that combine the efforts of the judiciary, probation, and treatment providers into a single intervention for offenders charged with or convicted of drug-related crimes. Drug courts provide supervision, drug testing, and treatment services. Drug courtfunding can come from various federal, state, county, and other sources. In fiscal year 2007, the Legislature appropriated $1 million in State General Fund monies for drug courts, of which $400,000 was allocated to juvenile drug courts.



Juvenile Justice Services Division

The AOC's Juvenile Justice Services Division (Division) is responsible for the administration of juvenile justice programs in coordination with the county juvenile courts. As of August 2007, the Division had a total of 39.6 positions, 5 of which were vacant. Almost one-third of the Division's positions (12.6 positions) are assigned to two units dedicated to developing, procuring, and monitoring treatment services.



State of Arizona

page



6



T Treatment Contracting and Evaluation (5 staff positions, 0 vacant)--This unit procures the state-funded treatment services and manages treatment provider contracts; evaluates state-funded treatment programs; and participates in interagency workgroups and committees. T Treatment Budget Monitoring and Program Development (7.6 staff positions, 0 vacant)--This unit manages state funding for the Juvenile Probation Services Fund and the Diversion, Family Counseling, and Juvenile Drug Court programs; monitors state-funded service provider contracts; handles interagency issues regarding juvenile services; and develops new treatment programs.



Scope and methodology

This audit focused on the Supreme Court's and counties' processes for identifying and addressing the needs of juveniles in the juvenile justice system and the AOC's contracting and oversight of juvenile treatment services vendors. The report presents findings and recommendations in the following areas: the direction of the Arizona Judicial Council, the AOC should work with the county juvenile courts to improve the processes that counties use to identify, plan for, and monitor the provision and effectiveness of treatment services to juveniles involved in the juvenile justice system.

Under



Supreme Court, through the AOC, uses a comprehensive contracting process, but it should strengthen its monitoring of juvenile treatment services vendors by conducting site visits for a sample of its smaller vendors and reviewing vendors' corrective action plans in a more timely manner.

The



The report also presents other pertinent information on the Interagency Integration Coordination Initiative, a state-wide effort to coordinate benefits for court-involved juveniles. Auditors used various methods to study the issues addressed in this report. These methods included interviewing Supreme Court and Maricopa and Pima County court and probation department staff; reviewing statutes; and reviewing Supreme Court and Maricopa and Pima County probation department policies and procedures. Additionally, the Supreme Court maintains two databases, both for the purposes of tracking juveniles in the juvenile justice system. The Juvenile Online Tracking System (JOLTS) is a state-wide juvenile probation and dependency management system, and the Juvenile Treatment (JTX) database tracks invoices that the AOC pays to vendors for the provision of treatment services. Auditors' review of the two databases included control work and limited testing. Auditors concluded that strong internal and



Office of the Auditor General

page



7



system controls were in place and the data was valid for use in drawing conclusions state-wide about juveniles and the treatment services provided to them. The databases do not include all information relevant to a juvenile's treatment. For example, Title XIX (Medicaid) funds expended on a juvenile are not tracked in the Court's databases, but are maintained by other agencies. In addition, auditors used the following specific methods: evaluate the processes that the Supreme Court and counties use to identify, plan for, and monitor juvenile treatment services, auditors reviewed and conducted analysis of the following: on some form of probation supervision, such as standard or intensive probation, during some point in fiscal year 2006, as reflected in a March 2007 download of the Court's JOLTS database;

Juveniles T o



random sample of 15 of the 3,995 Maricopa County juvenile standard probation files that were active between July 1, 2005 and June 30, 2006, consisting of 5 files from Maricopa County's Durango office, 5 files from the Southeast Facility office, 2 files from the West Valley office, 2 files from the Sunnyslope office, and 1 file from the Southport office; and

A



random sample of 10 of the 833 Pima County juvenile standard probation files that were active between July 1, 2005 and June 30, 2006, consisting of 8 files from Pima County's main office and 2 files from its Northwest office.

A



Finally, auditors interviewed six Maricopa and Pima County probation officers and reviewed literature and national standards on juvenile justice best practices in assessing risk and needs, developing case plans, and monitoring progress from national organizations (see Bibliography, page 37, for a listing of the literature reviewed). Auditors also reviewed policies, practices, risk and needs assessments, and case planning requirements from Utah, Washington, and Florida.1 evaluate the AOC's contracting and monitoring processes, auditors reviewed and analyzed a random sample of 16 of the 51 vendor monitoring files for fiscal years 2005 and 2006 and a sample of 9 of the 19 vendor selfaudits performed in calendar year 2006, observed a monitoring site visit conducted in June 2007, analyzed fiscal year 2007 treatment services expenditure data from the JTX database, and reviewed a 2001 AOC internal audit and judicial branch procurement rules.

T o



1



Auditors contacted Utah based on an Arizona juvenile court official recommendation. Utah juvenile court personnel then referred auditors to Florida and Washington, since both Florida and Utah have in part based their juvenile assessment tools on Washington's juvenile assessment tool.



State of Arizona

page



8



obtain information on the Interagency Integration Coordination Initiative, auditors interviewed two Initiative participants and reviewed information from the Governor's Office for Children, Youth and Families Web site and Initiative documents such as the letter of agreement, the Framework for Interagency Practice Protocol, and a potential outcomes document. provide information for the report's Introduction and Background section, auditors reviewed various AOC documents and reports, including documents related to the Standardized Program Evaluation Protocol Initiative and the Juveniles Processed in the Arizona Court System FY2006 report; information from the Supreme Court's and other state agencies' Web sites; budgetary information provided by the AOC; the Juvenile Delinquency Guidelines and other information from the National Council of Juvenile and Family Court Judges; and AOC and juvenile county court policies and procedures; and analyzed state-wide data from the JOLTS system for 2006 and JTX database for fiscal year 2007. This audit was conducted in accordance with government auditing standards. The Auditor General and staff express appreciation to the Chief Justice and the Director and staff of the Administrative Office of the Courts and to the presiding juvenile court judges and directors, and the probation personnel at the counties auditors visited for their cooperation and assistance throughout the audit.

T o



T o



Office of the Auditor General

page



9



State of Arizona

page



10



FINDING 1

Supreme Court should improve assessment, planning, and monitoring processes

The Supreme Court (Court) should take steps to improve the processes that counties use to identify, plan for, and monitor the provision and effectiveness of services to juveniles. To help determine a juvenile's risk for re-offending and need for services, the Court has developed standardized risk and needs assessment tools. However, based on auditors' review of state-wide probation data and a random sample of case files from Maricopa and Pima Counties, these assessments were not always completed or completed in a timely manner. Additionally, although the Court requires probation officers to develop case plans, it does not have policies and procedures that guide how these plans should be developed or used. Finally, under the direction of the Arizona Judicial Council, the Administrative Office of the Courts (AOC) should work with the county juvenile courts to develop and implement policies and procedures for monitoring juveniles' progress against case plans and reassessing and updating case plans based on this progress.



Supreme Court's risk- and needs-assessment tool

As required by statute, the Court has developed a formal process and instruments to address the risks and needs of delinquent and incorrigible youth and their families. Specifically, A.R.S. �8-246(C) requires the use of a "common risk needs assessment instrument to be used for each juvenile who is referred to the juvenile court. The juvenile court shall update the risk needs assessment on each subsequent referral of the juvenile to the juvenile court, and the court shall use the risk needs assessment to determine the appropriate disposition of the juvenile." Originally developed and implemented in 1994, the Court revised its risk and needs assessment instruments in 2000 and 2002, respectively, to assess referred juveniles. These assessment instruments consist of three separate assessments, which are completed by a probation officer and designed to help determine if a juvenile will re-offend and assist in identifying and prioritizing the specific needs of the juvenile and family. Additionally, these instruments were intended to help probation officers develop a practical case plan for the juvenile. These instruments include the following:



The Court revised its risk and needs assessment instruments in 2000 and 2002.



Office of the Auditor General

page



11



R Risk Assessment--This instrument consists of ten questions designed to assess the likelihood of re-offense and determines questions to complete on the needs assessment. The questions attempt to gather information on risk factors, such as family conflict, substance abuse, school attendance, and mental health. Based on the responses to these questions, the assessment produces a risk score that determines the juvenile's likelihood of re-offending within the next year compared to a juvenile with similar risk factors. For example, a score of 76 indicates that 76 percent of juveniles with similar answers re-offend within 1 year. The Court requires that this assessment be completed each time a juvenile is referred to juvenile court. The risk assessment instrument was validated in 1998 by LeCroy and Milligan Associates, Inc., and again in 2007 by Columbia University.1 Disposition--Date that a delinquent, incorrigible, or dependent youth receives consequences from the court.

Source: Arizona Supreme Court's Juvenile Justice Services Division Web site.



N Needs Assessment--This instrument consists of 13 needs categories designed to assess the needs of juveniles and their families and develop broad goals that identify potential areas of treatment. The 13 categories correspond to some questions on the risk assessment, including mental health, substance abuse, educational functioning, and family functioning. The needs assessment should provide the probation officer with a brief outline of the juvenile's serious issues and should be used to help develop an appropriate case plan. The Court requires completion of this assessment for all adjudicated youth prior to disposition. S Strengths/Protective Factors Assessment--This assessment consists of 12 questions that assess juveniles' strengths and identify factors that can reinforce positive behavior. The 12 questions relate to protective factors, which are positive influences and traits that may help prevent youth from further court involvement. Identification of strengths offers the opportunity to focus on positive areas in a juvenile's life and potentially reduce the likelihood of further involvement in the justice system. The Court does not require completion of this assessment. Probation officers complete these assessments using the Court's Juvenile On-Line Tracking System (JOLTS) system.2 After completing these assessments, the probation officer can generate a JOLTS report that summarizes the information from these assessments. Referred to as an Identified Needs Report, this report consists of information such as the risk score, a list of identified and prioritized needs, goals generated to address risk and needs, a list of further assessments to conduct on the juvenile, and a list of the juvenile's strengths. The probation officer should use this information to help develop a case plan.



The Court's strengths/protective factors assessment assesses juveniles' strengths and identifies factors that can reinforce positive behavior.



1



The Revalidation of the Arizona Risk/Needs Assessment Instrument was prepared for the Supreme Court by Craig Schwalbe, Ph.D., with the Columbia University School of Social Work in October 2007. JOLTS is a state-wide juvenile probation and dependency management system that provides several functions to users, including managing probation caseload and detention, tracking billing and financial information, conducting risk/needs assessments, and tracking treatment services.



2



State of Arizona

page



12



Risk assessment not fully used

Under the direction of the Arizona Judicial Council, the AOC should take steps to ensure that its risk assessment instrument is completed as required. The Supreme Court and county juvenile probation departments require the risk assessment to be completed for all youth referred to juvenile court. However, auditors' review of state-wide data for juveniles on probation in fiscal year 2006 and a case file review of a random sample of 25 juveniles on probation in fiscal year 2006 from Maricopa and Pima Counties determined that these assessments were not always completed or completed in a timely manner. Therefore, under the direction of the Arizona Judicial Council, the AOC should work with the county juvenile courts to develop and implement policies and procedures regarding when and how its risk assessment should be completed. The Arizona Judicial Council assists the Supreme Court and the chief justice in developing and implementing policies and procedures for the administration of all courts. For example, it studies the internal operation of the courts and plans for future developments. It also promotes improvements and responds to issues concerning judicial administration by reviewing and recommending for adoption by the chief justice proposed administrative orders, code sections, rules, and policies.



Counties inconsistently complete risk assessments--The Court requires

counties to complete a risk assessment each time a youth is referred to the county juvenile courts. Additionally, statute and Maricopa and Pima County juvenile court policies indicate that the assessment should be used to determine the disposition or outcome for a juvenile. However, some juveniles do not receive risk assessments upon referral to the county juvenile court, and some assessments are not completed prior to disposition. Specifically, according to JOLTS information, for the 12,591 juveniles on probation in fiscal year 2006, 95 percent had at least one completed risk assessment. However, most of the 12,591 juveniles had multiple referrals and 3,152, or 25 percent, of these juveniles had at least one referral for which a risk assessment was not completed, and 8,484, or more than 67 percent, of these juveniles had at least one referral where a risk assessment was not completed prior to disposition.1 Additionally, auditors' review of the random sample of 25 juveniles' case files (15 juveniles from Maricopa County and 10 juveniles from Pima County) determined that 19 of these 25 juveniles had at least one referral for which a risk assessment was not completed prior to disposition. Auditors' review of these same 25 juveniles' files also noted that 12 juveniles had a risk assessment that covered more than one referral. Although covering multiple referrals in a single risk assessment may be appropriate when these referrals are processed at the same disposition hearing, auditors noted instances where the disposition hearings occurred on different dates for bundled referrals. For example, one juvenile had a risk assessment completed on March 14, 2006, that was credited in JOLTS as having been conducted for three separate referrals that had disposition hearings on January 24, 2005, January 3, 2006, and April 17, 2006. In addition to completing a risk assessment for referrals that had disposition hearings nearly 15 months apart, the risk assessment was not completed prior to the juvenile's disposition hearings for the first two referrals.

1 For this analysis, Auditor General staff excluded administrative referrals created by the courts for various purposes, referrals handled in courts lower than the Superior Court, and referrals made prior to July 1, 2000 (the implementation date of the revised risk-assessment tool). Office of the Auditor General

page



According to JOLTS, 25 percent of juveniles on probation in fiscal year 2006 had at least one referral for which a risk assessment was not completed.



13



Without completed risk assessments, the Court lacks important information to determine the appropriate level of supervision and treatment services for juveniles on probation. According to guidance provided by the National Council of Juvenile and Family Court Judges, risk assessment scores can be used for intake diversion decisions and court dispositions, and to determine the level of supervision required for youth on probation.1 Risk assessment scores can be used to classify youths into groups with different recidivism probabilities, and groups with higher probability rates should receive more supervision and more services. According to the National Center for Juvenile Justice's Desktop Guide to Good Juvenile Probation Practice (Desktop Guide), some probation departments attempt at intake to identify those youth who are at risk of becoming chronic offenders so that they can be targeted for early intervention.2 For example, the Orange County California Probation Department identifies potential chronic offenders who share three or more profile factors and targets these cases for more aggressive, familyfocused services. The Desktop Guide further states that the identification of risk and protective factors makes effective delinquency prevention a practical possibility.



AOC should improve use of risk assessment instrument--Under the

Under the direction of the Arizona Judicial Council, the AOC should work with the county juvenile courts to improve the use of its risk assessment.



direction of the Arizona Judicial Council, the AOC should work with the county juvenile courts to improve the use of its risk assessment. First, it should provide guidelines to the county juvenile courts and probation officers regarding the time frame in which a risk assessment must be completed. If the juvenile's outcome is determined prior to the disposition hearing, for example, at the adjudication hearing, the risk assessment would not be done and available for the Court to review. Second, the AOC should provide guidance on when, and under what circumstances, one assessment may be performed to address multiple referrals. Probation officers indicated that it may not make sense to complete separate assessments for each referral when the referrals occur relatively close to one another since the referrals may be processed at the same time in court. Third, the AOC should develop policies and procedures for using a juvenile's risk assessment to recommend to the judge an appropriate level of supervision for juveniles on probation and to help determine who should receive more aggressive treatment services. Finally, similar to Washington and Utah, the AOC should use the risk assessment to determine which juveniles should receive needs and/or strengths/protective factors assessments. For example, Utah and Washington conduct a needs and strengths assessment for those juveniles who are at moderate and high risk to re-offend, as determined by the risk assessment.



1 2 State of Arizona

page



Wiebush, Richard G. Graduated Sanctions for Juvenile Offenders: A Program Model and Planning Guide. Reno, NV: National Council of Juvenile and Family Court Judges. Juvenile Sanctions Center, 2003. National Center for Juvenile Justice. Desktop Guide to Good Juvenile Probation Practice. Pittsburgh, PA: National Center for Juvenile Justice, 2002.



14



Needs and strengths/protective factors assessments not fully used

Similar to its risk assessment, the AOC should make various improvements to its needs assessment instrument. Although the Court and county juvenile courts require the completion of the needs assessment for all adjudicated juveniles, these assessments were not always completed or completed in a timely manner. Maricopa and Pima Counties' probation officers often use other needs-assessment tools instead of the Court's needs assessment instrument. In addition, the Court's strengths/protective factors assessment is not completed for most juveniles. Therefore, the AOC should continue its efforts to evaluate and revise or redo its needs assessment tool to better meet best practices and to provide a useful and easy tool for probation officers; establish a policy to require the completion of a strengths/protective factors assessment; and establish time frames for completing these assessments.



Some needs assessments not completed--The



Court requires the counties to complete its needs assessment for all juveniles who have been adjudicated by the juvenile court. Additionally, statute requires that the needs assessment should be used to determine the appropriate disposition of the juvenile. As a result, it should be completed prior to disposition. However, the counties are not consistently completing these needs assessments as required. According to JOLTS information, for the 12,591 juveniles on probation in fiscal year 2006, 5,783, or nearly 46 percent, had at least one instance for which a needs assessment was not completed, and 5,008, or nearly 40 percent, had at least one instance where a needs assessment was not completed prior to disposition.1 However, the JOLTS data reflects only the use of the Court's needs assessment and does not reflect any alternative needs assessments that the counties may have completed. Auditors' review of the case files for the random sample of 25 juveniles suggests that most juveniles receive a needs assessment of some type. Auditors found that 18 of these 25 juveniles had at least one instance for which the Court's needs assessment was not completed prior to disposition. After factoring in alternative needs assessments, only 3 juveniles had at least one instance of not having any type of needs assessment completed prior to disposition.

According to JOLTS, nearly 46 percent of the juveniles on probation in fiscal year 2006 had at least one instance for which a needs assessment was not completed.



Counties using other tools or methods to assess needs--According to

seven of the eight juvenile probation officers and supervisors from Maricopa and Pima Counties who auditors interviewed, the Court's needs assessment is not useful because it provides little new information or guidance in writing a disposition report or developing a case plan. Specifically, the probation officers already know the juvenile's needs from contacting interested parties for information (school, counselors, etc.) and completing other tools. For example, according to a

1 Auditor General staff limited the analysis to referrals that resulted in a disposition to a probation status and also excluded administrative referrals created by the courts for various purposes, referrals handled in courts lower than the Superior Court, and referrals made before October 1, 2002 (the implementation date of the needs assessment tool). Office of the Auditor General

page



15



Maricopa County juvenile court official, their procedures require probation officers to complete Family Social forms when juveniles are initially adjudicated to gather historical information about the juvenile and family. This includes information about the juvenile's family history and background, school status, criminal history, and drug use. In addition, both counties gather information from interviews with the juvenile and family, meetings with relevant parties such as school officials and service providers, and psychological evaluations and other professional assessment tools for juveniles determined to have substance abuse or sex offender issues or when ordered by the Court.



Strengths/protective factors assessment not typically used--Auditors

also noted that when the counties completed the Court's needs assessment, they generally did not complete the Court's correlating strengths/protective factors assessment. Although probation officers are not required to complete a strengths/protective factors assessment, the Court developed this assessment to help with developing goals and creating case plans for juveniles. Auditors' review of the files for 25 juveniles on probation determined that 23 of the 25 juveniles did not have a completed Court strengths/protective factors assessment. According to the National Council of Juvenile and Family Court Judges, it is important to use a formal, structured assessment of both needs and strengths as the foundation for the case plan.1 Additionally, the Desktop Guide recommends assessing protective factors and strengths in such areas as family and parenting, the juvenile's attributes, and school, and determine what strengths can be built upon.2



Needs assessment being evaluated for possible revisions--The AOC

believes that a state-wide standardized risk and needs assessment tool is required to not only meet statutory requirements, but as a formal process to better ensure that juveniles are receiving appropriate treatment services. As a result, the AOC established a Needs Workgroup consisting of representatives from 11 county juvenile courts, the AOC, Arizona State University, and the Arizona Department of Juvenile Corrections in September 2007 to determine if its risk/needs assessment conforms to best practices from other states and agencies, or if assessment tools from another state or agency can better meet its needs. As of October 2007, the AOC had visited Utah to learn about its needs assessment and integrated approach, and contacted Florida to obtain information regarding its needs assessment. The AOC's effort is consistent with best practices that suggest the use of consistent, structured tools and/or approaches to assess needs and strengths. Specifically, according to the National Council of Juvenile and Family Court Judges, it is important to use a formal, structured needs/strengths assessment as the foundation for the case plan and to ensure greater consistency in the

1 Wiebush, Richard G. Graduated Sanctions for Juvenile Offenders: A Program Model and Planning Guide. Reno, NV: National Council of Juvenile and Family Court Judges, Juvenile Sanctions Center, 2003. National Center for Juvenile Justice. Desktop Guide to Good Juvenile Probation Practice. Pittsburgh, PA: National Center for Juvenile Justice, 2002.



The AOC's Needs Workgroup is reviewing the Court's risk and needs-assessment tool



2



State of Arizona

page



16



assessment process.1 The National Council further states that to avoid inconsistent decisions that come from informal, subjective assessments, highly structured assessment instruments should be used so that the same basic set of questions are asked of all youth, and decision rules should be used so that assessment results lead directly to an indicated decision. Additionally, using assessment procedures that do not result in the right juvenile's being consistently linked to the right intervention may lead to increased risk to public safety, inefficient use of resources, and inequitable treatment placements among youths. Finally, the Desktop Guide lists different types of assessments as part of a predisposition investigation, including risk, needs, and strengths, to assist with disposition decision-making and determining how best to meet the juvenile's and community's needs.2 Completion of these assessments prior to disposition help determine what level of security or supervision for the juvenile will be necessary to keep the community safe, and what measures will enable the juvenile to lead a more lawabiding, pro-social life. Therefore, the AOC should continue its efforts to evaluate and revise its needs assessment to better meet best practices and to provide a useful and easy tool for probation officers. Under the direction of the Arizona Judicial Council, the AOC should also work with the county juvenile courts to evaluate its approach for assessing strengths/protective factors and ensure that this approach is consistent with best practices, and establish policy requiring the completion of a strengths/protective factors assessment.



Case planning needs improvement

Under the direction of the Arizona Judicial Council, the AOC should also take steps to improve the case plans developed for juveniles involved in the juvenile justice system. Although the Court intended to have the counties develop case plans based on the risk and needs assessments, it has never implemented standards, policies, and procedures for the completion and use of juvenile case plans.



Case plan development and use need improvement--In



the early 2000s, one of the Court's goals for its risk and needs assessments was to generate broad goals for juveniles based on identified needs, further assessment(s) required, and the probation officer comments, which would then be incorporated into case plans. In addition to including these goals and the identified needs, the Court intended that case plans would also identify any further assessment(s) required and include probation officer comments. According to an AOC manager, although the AOC established a workgroup in 2003 to develop a model case plan that would flow directly from the risk and needs assessments, this project was



The AOC's efforts to develop a model case plan were discontinued.



1



Wiebush, Richard G. Graduated Sanctions for Juvenile Offenders: A Program Model and Planning Guide. Reno, NV: National Council of Juvenile and Family Court Judges, Juvenile Sanctions Center, 2003. National Center for Juvenile Justice. Desktop Guide to Good Juvenile Probation Practice. Pittsburgh, PA: National Center for Juvenile Justice, 2002. Office of the Auditor General

page



2



17



discontinued because of the redirection of programming resources to the rewriting of JOLTS. Although the Court requires probation officers to develop case plans, it does not have policies and procedures that guide how these plans should be developed or used. For the two counties that auditors reviewed, the development and use of case plans differ. Specifically: M Maricopa County juvenile court case plans--Maricopa County juvenile court policies require the completion of a formal case plan for juveniles who are placed on probation. According to policy and a county supervisor, case plans should be completed within 45 days of a juvenile's being placed on probation and should be monitored by the probation department. However, despite this policy, auditors found that for the 15 juveniles reviewed, in 14 of the 27 instances where these juveniles were adjudicated and placed on probation, a formal case plan was not developed. In addition, when formal case plans were completed, they did not always address all factors, such as specific goals and objectives, action-oriented steps to complete objectives, or time frames to accomplish goals or complete services. According to the Desktop Guide, a case or supervision plan should outline clear goals and meaningful objectives for the juvenile to achieve while supervised, detail activities to accomplish the goals and objectives, and establish time frames for completing each objective.1 Although Maricopa County Juvenile Court officials indicated that probation officers receive some guidance and training on case plan development, auditor review of case plans found that they often lacked some of these essential elements.



Case plans should include goals, objectives, action items, and time frames.



P Pima County Juvenile Court case plans--Pima County's probation manual requires probation officers to include a case plan section in the juvenile's disposition report that provides information on the probation officer's evaluation of the case and the main issues to be addressed. However, the County has not developed policies and procedures consistent with best practices to guide probation officers on the information that should be included in this section of the disposition report. As a result, although probation officers completed this section of the court report for all ten Disposition Report--Probation officers' juveniles, this section often lacked the same essential elements that were written report to the Court with detailed information regarding the juvenile and missing in Maricopa County's case plans. Without case plans that include his/her family. Contains a case evaluation, essential elements such as goals, objectives, action steps, and time the main issues to be addressed, the plan frames, the Court and probation officers lack a blueprint to guide the for the juvenile, and how the plan supervision of juveniles. According to the Desktop Guide, although addresses each issue. This report is sent probation officers base supervision activities on the court-ordered to the judge prior to disposition. conditions of probation, these conditions by themselves do not provide Source: Pima County Juvenile Court probation manual. the guidance required for good supervision practice.2 The Desktop Guide further states that the supervision or case plan can provide this guidance

1 National Center for Juvenile Justice. Desktop Guide to Good Juvenile Probation Practice. Pittsburgh, PA: National Center for Juvenile Justice, 2002. National Center for Juvenile Justice. Desktop Guide to Good Juvenile Probation Practice. Pittsburgh, PA: National Center for Juvenile Justice, 2002.



2 State of Arizona

page



18



and should specify the level of supervision required to address the risk the juvenile poses to the community, and how a juvenile's day will be structured in productive activities; how the accountability requirements will be fulfilled; and address the behavior problems, thinking errors, or skill deficits that place the juvenile at greatest risk for continued criminal activity and specify the services/interventions that will address these needs.



AOC should improve case planning--Under



the direction of the Arizona Judicial Council, the AOC should work with the county juvenile courts to develop and implement policies and procedures guiding the creation of case plans for juveniles on probation. Further, the AOC and county juvenile courts should develop these policies and procedures using best practices from national organizations and other states. For example, the Desktop Guide states that a case plan should have specific goals, measurable behavioral objectives, action steps, and completion dates; and that probation officers should set priorities by reviewing the assessments and considering the three factors that place the juvenile at the greatest risk for continued delinquent behavior.1 Additionally, federal guidance states that case plans should clearly identify goals and intervention priorities, and that probation officers should specify rewards and sanctions for complying or not complying with agreed-upon behaviors. This guidance further states that case plans must be flexible and responsive, and should be reviewed approximately every 2 to 3 months to assess progress toward completing stated objectives.2 Utah requires the development of case plans for all juveniles on probation and uses two worksheets to create case plans from the completed risk and needs assessment. These worksheets guide probation officers in determining and prioritizing youths' needs and how to use this information to develop an appropriate case plan. Florida requires the development of case plans for all juveniles on probation, and these case plans should include target dates for each goal and action step in the case plan. The juvenile and parents/guardians must sign and receive a copy of the case plan and be informed of the consequences of failing to fulfill the plan's goals. Under the direction of the Arizona Judicial Council, the AOC should also work with the county juvenile courts to define an appropriate time frame for completing a case plan. For example, case plans must be completed within 30 days of the order for probation according to Utah's policy, and within 14 days of placement on supervision according to Florida's policy.



1 2



National Center for Juvenile Justice. Desktop Guide to Good Juvenile Probation Practice. Pittsburgh, PA: National Center for Juvenile Justice, 2002. U.S. Department of Justice. Guide for Implementing the Comprehensive Strategy for Serious, Violent, and Chronic Juvenile Offenders. Washington, D.C: U.S. Department of Justice, Office of Justice Programs, Office of Juvenile Justice and Delinquency Prevention, May 1995.



Office of the Auditor General

page



19



AOC and counties should improve treatment services monitoring

To help ensure that county juvenile probation officers appropriately complete required assessments and develop and review juveniles' progress against case plans, the AOC and counties should improve their monitoring efforts. Counties inconsistently monitor completion of juveniles' treatment services and do not consistently reassess juveniles' needs based on their progress or to update case plans. As a result, juveniles may not receive treatment services in a timely manner or may not receive treatment services that address their identified needs. Therefore, under the direction of the Arizona Judicial Council, the AOC should work with the county juvenile courts to develop and implement policies and procedures for monitoring timely completion of services, measuring a juvenile's progress with reassessments, and updating case plans based on this progress. Further, the AOC should expand its Operational Review process to monitor compliance with assessment and case-planning policies.

Monitoring of treatment services provided to juveniles in Maricopa and Pima Counties is inconsistently performed.



Inconsistent monitoring of services--Although



some probation officers in Maricopa and Pima Counties monitor juveniles' progress, treatment monitoring is not consistently performed by all probation officers in the two counties reviewed. Specifically, auditors' review of the sample of the case files for 25 Maricopa and Pima County juveniles on probation found that 5 of these juveniles had not received treatment services as ordered by the juvenile court, services that the court typically funds. Additionally, although Maricopa and Pima Counties track juveniles' progress through case notes, auditors' review showed that 24 of the 25 juveniles' case files contained inconsistent documentation on whether provided services successfully addressed identified needs. For example, these cases had some instances where service completion and progress were not clearly indicated in case notes. Finally, AOC officials indicated that the AOC does not centrally monitor the completion of treatment services received through funding sources other than the Supreme Court. According to AOC and Maricopa and Pima County court officials, it is difficult to obtain treatment information from Medicaid providers as these providers have not contracted with the Court to provide Medicaid services and are under no obligation to provide treatment service information to the Court. As a result, counties do not always know if juveniles received needed services. The following case example illustrates a juvenile that has received some, but not all, of the services ordered by a juvenile court judge. a 16-year-old juvenile, was first referred to juvenile court for vehicle burglary in July 1999, when he was 8 years old. He was placed on probation and ordered to complete community work hours and 7 hours of counseling. By March 2007, he had been referred to the juvenile court nine additional times, including six times for theft/shoplifting, twice for criminal damage (once for five felony counts of criminal damage between $2,000 and $9,999), and once for intent to cause injury and trespassing. A psychological evaluation completed in September 2004 indicated mental health concerns, including

James,



State of Arizona

page



20



depression and an adjustment disorder, and family and school issues. In September 2006, he received his first court needs assessment, which stated that he needed a more detailed substance or alcohol abuse assessment, and that he was experimenting with alcohol, and indicated concerns with drug use. However, 8 years after first becoming involved with the juvenile court system, James has received some, but not all, of the treatment services ordered by the Court or indicated by assessments. Specifically, he has received 7 hours of counseling four different times in response to four separate referrals (1999, 2003, 2004, and 2006), and the same shoplifting class three different times in response to three separate referrals (2003, 2004, and 2006). He did not receive court-ordered drug testing in 2004 or a drug substance or alcohol abuse assessment as indicated on his 2006 needs assessment. Another means of monitoring or measuring juveniles' progress is through conducting reassessments. However, according to probation department officials, neither Maricopa nor Pima County have policies and procedures for using assessment instruments to measure juveniles' progress and/or reassess risk levels. Auditors' review of the sample of 25 Maricopa and Pima County juvenile case files found no documentation indicating that probation officers use the Court's risk, needs, or strengths/protective factors assessments to reassess a youth's progress and continuing needs.



AOC and counties should improve monitoring and make adjustments as needed--Under the direction of the Arizona Judicial

Council, the AOC should work with the county juvenile courts to develop and implement policies and procedures for monitoring the timely completion of services. The National Council of Juvenile and Family Court Judges states in its Juvenile Delinquency Guidelines that dispositions will only be effective if the juvenile delinquency court ensures that the youth, parents, and probation follow through with court orders and that service providers follow through with timely, necessary services.1 The Desktop Guide also recommends that probation officers track a juvenile's progress by regularly monitoring compliance and performance.2 For example, probation officers can track a juvenile's progress on a practical rehabilitation goal involving attendance and successful completion of treatment classes and programs. Case notes should document the youth's behavior as well as the probation officer's efforts to implement the case plan. The Desktop Guide also recommends that probation officers request and maintain periodic written reports from those providers involved with the juvenile regarding the juvenile's status in complying with case plan objectives. Therefore, under the direction of the Arizona Judicial Council, the AOC should work with the county juvenile courts to establish policies and procedures that direct juvenile probation officers in all counties to monitor the delivery of treatment services received by juveniles regardless of the funding source to help ensure that treatment services are addressing a juvenile's needs in a timely manner.

1 2 Grossmann, David E. and Maurice Portley. Juvenile Delinquency Guidelines: Improving Court Practice in Juvenile Delinquency Cases. Reno, NV: National Council of Juvenile and Family Court Judges, 2005. National Center for Juvenile Justice. Desktop Guide to Good Juvenile Probation Practice. Pittsburgh, PA: National Center for Juvenile Justice, 2002. Office of the Auditor General

page



21



Policies and procedures needed that direct probation officers to periodically reassess juveniles' risk and needs.



As part of its monitoring efforts and under the direction of the Arizona Judicial Council, the AOC should also work with the county juvenile courts to develop and implement policies and procedures that direct county juvenile probation officers to measure juveniles' progress by periodically reassessing their risks and needs and making adjustments to the case plan as necessary. The federal Office of Juvenile Justice and Delinquency Prevention Guidelines state that regular reassessment of risk and needs factors, which is generally completed every 90 days, is an integral part of case management and will help determine how to adjust case plans and supervision requirements.1 Utah's policy states that its full assessment should be used to reassess needs every 90 days and after new offenses to measure a juvenile's progress toward becoming low risk. Florida's probation manual describes its case plans as "living" documents that must be reviewed by supervisors and updated at least every 90 days to reflect when sanctions, goals, and action steps are completed and when new court orders or voluntary goals are added. The manual also states that a youth's level of supervision is reassessed at a meeting every 90 days and after new violations, and that the risk/needs reassessment must be completed prior to that meeting for youth identified as moderate-high and high risk to re-offend. Finally, many of the recommendations made to improve the assessment, planning, and monitoring of juveniles in the court system and the services they receive will require changes to policies, procedures, and assessment tools. Therefore, once these policies and procedures are developed, the AOC should submit them to the Arizona Judicial Council for review and approval.



AOC should expand reviews of counties to monitor compliance with policies--To help ensure that counties appropriately implement its policies and

procedures regarding treatment services, the AOC should expand the scope of its juvenile county court operational reviews. The AOC conducts operational reviews of county juvenile courts once every 5 years. During these reviews, the AOC reviews the county juvenile court's operations to determine whether court policies and procedures are being followed. For example, the AOC uses JOLTS reports to determine whether risk and needs assessments have been completed. However, these reports do not include information on whether the assessments are completed in a timely manner. Further, these reviews should be expanded to include a review of county compliance with the additional policies, procedures, and practices recommended in this report once the Court implements them.



Recommendations:

1. The AOC should seek the Arizona Judicial Council's direction regarding the need to develop and implement policies and procedures related to the risk and



1



U.S. Department of Justice. Guide for Implementing the Comprehensive Strategy for Serious, Violent, and Chronic Juvenile Offenders. Washington, D.C.: U.S. Department of Justice, Office of Justice Programs, Office of Juvenile Justice and Delinquency Prevention, May 1995.



State of Arizona

page



22



strengths/protective factors assessments, case planning, and monitoring the delivery of treatment services to juveniles, as necessary. 2. If the Arizona Judicial Council approves the need to develop and implement policies and procedures, the AOC should work with the county juvenile courts to develop policies and procedures regarding the completion of the risk assessment. These policies and procedures should: a. Prescribe the time frame for completing this assessment and allow for instances to potentially complete one risk assessment if multiple referrals occur within a specified time frame; Recommend how a juvenile's risk assessment should be used to determine an appropriate level of supervision for juveniles on probation and if the juvenile should receive more aggressive treatment services; and Determine whether the juvenile should receive needs and/or strengths/protective factors assessments based on the juvenile's risk assessment.



b.



c.



3.



The AOC should continue its efforts to evaluate and revise its needs assessment to better meet best practices and to provide a useful, easy tool for probation officers. If the Arizona Judicial Council approves the need to evaluate the approach for assessing strengths/protective factors, the AOC should work with the county juvenile courts to ensure this approach is consistent with best practices and establish policy requiring the completion of a strengths/protective factors assessment. If the Arizona Judicial Council approves the need to develop and implement policies and procedures guiding the creation of case plans for juveniles, the AOC should work with the county juvenile courts to develop and implement such policies and procedures for case plans which are consistent with best practices. These policies and procedures should require that case plans: a. b. c. Address prioritized needs as identified in the risk and needs assessments; Include action steps for reaching defined goals and objectives; Use target dates for action steps, goals and objectives, and completion of services; and Receive regular supervisory review.



4.



5.



d.



Office of the Auditor General

page



23



6.



If the Arizona Judicial Council approves the need to define an appropriate time frame for completing a case plan, the AOC should work with the county juvenile courts to define an appropriate case plan completion time frame. If the Arizona Judicial Council approves the need to develop polices and procedures regarding monitoring the delivery of treatment services to juveniles, the AOC should work with the county juvenile courts to establish policies and procedures that: a. Direct juvenile probation officers to monitor the delivery of treatment services received by juveniles, regardless of the funding source, to help ensure treatment services are addressing a juvenile's needs; and Require county juvenile courts to periodically reassess a juvenile's risk and needs, and adjust case plans as needed.



7.



b.



8.



Once the various policies and procedures recommended in this audit report are developed, the AOC should submit these policies and procedures to the Arizona Judicial Council for its review and approval. The AOC should conduct a sufficient level of analysis during its county juvenile court Operational Reviews to determine whether assessments are conducted within the required time frames. It should also expand the reviews to include a review of county compliance with the additional policies, procedures, and practices recommended in this report once they have been implemented.



9.



State of Arizona

page



24



FINDING 2

Contracting process thorough, but AOC should improve monitoring of treatment services vendors

The Supreme Court, through the Administrative Office of the Courts (AOC), has established a thorough process to contract with treatment services vendors, but it should take some steps to improve its vendor monitoring. The AOC is responsible for the administration and management of contracts to provide treatment services to court-involved juveniles. To contract for these treatment services, the AOC has developed and follows a comprehensive process. Additionally, the AOC monitors vendor compliance with contract terms, but should improve various features of its monitoring process, including modifying its vendor monitoring site visit sampling approach to conduct site visits for a sample of its smaller vendors and reviewing and approving vendor corrective action plans in a timely manner.



AOC administers and manages treatment services contracts

The AOC administers contracts for 17 treatment services categories for juveniles in the juvenile justice system. These services include out-of-home care, such as group homes; professional evaluations, such as psychological evaluations; outpatient mental health services, such as individual or family counseling; and education services, such as General Equivalency Diploma tutoring and testing. The AOC pays for these services with State General Fund monies that have been appropriated to the Juvenile Probation Services Fund (JPSF), the Family Counseling Program, and other funds. As authorized by A.R.S. ��8-321 and 8-322, the Supreme Court and counties can use JPSF monies to reduce the number of repetitive juvenile offenders by providing various services, including treatment services. A.R.S. ��8261(2) and 8-264(A) authorize the Supreme Court and counties to use Family Counseling Program monies to strengthen family relationships and prevent juvenile

The AOC administers contracts for 17 treatment services categories.



Office of the Auditor General

page



25



delinquency. For fiscal year 2007, the AOC contracted with 161 treatment services vendors. A total of 151 vendors billed services totaling nearly $22.8 million to the AOC. As illustrated in Table 2, for fiscal year 2007, juveniles in Maricopa County received more than $11.4 million in Supreme Court-funded treatment services, while juveniles in Pima County received nearly $3.8 million in Supreme Court-funded treatment services. In fiscal year 2007, juveniles in Apache, La Paz, and Greenlee Counties each received less than $100,000 in Supreme Court-funded treatment services.



Table 2:



Expenditures for Supreme CourtFunded Juvenile Treatment Services By County Fiscal Year 2007 (Unaudited)

$11,437,464 3,771,618 756,812 714,793 706,845 635,018 562,275 312,915 281,614 207,451 140,315 112,928 90,576 63,790 35,434 $19,829,8481



Supreme Court has comprehensive contracting process

The Supreme Court has developed and uses a comprehensive process to contract for juvenile treatment services. Specifically, the Supreme Court has adopted various procurement rules for the judicial branch, and the AOC has adopted various procurement practices that consist of the following steps: C Creating a qualified vendor list--Initially, the AOC creates a qualified vendor list that consists of vendors who meet the Supreme Court's minimum requirements, such as having accreditation from a national accreditation organization or having been prequalified by the AOC. The AOC recognizes accreditation from the following three organizations: the Joint Commission Accreditation of Health Organizations (JCAHO), Council on Accreditation (COA), or the Commission on the Accreditation of Rehabilitation Facilities (CARF). If a vendor is not accredited, the AOC prequalifies the vendor by assessing its ability to meet minimum standards. These standards include legal authorization to conduct business in Arizona, the ability to meet insurance requirements, fiscal stability, general management requirements, and the capacity to manage and monitor contract requirements.



Maricopa Pima Pinal Yavapai Yuma Cochise Mohave Coconino Gila Navajo Graham Santa Cruz Apache La Paz Greenlee Total

1



This total does not include the more than $2.94 million used by the Supreme Court for multiple-county treatment services. Auditor General staff analysis of fiscal year 2007 expenditure information from the Supreme Court's juvenile treatment services invoice tracking system and interviews with the AOC staff.



Source:



The qualified vendor list consists of either independent practitioners or full procurement providers/vendors. An independent practitioner is either the sole owner of a business or a legal partner in a business where one owner/partner or both partners are the proposed service delivery professionals. A full procurement vendor is an accredited agency or business where more than one individual delivers the proposed services. Independent practitioners need only to register with the AOC to be placed on the vendor list. The JCAHO, COA, or CARF must accredit full procurement vendors, or they must undergo the AOC's prequalification to be placed on the vendor list.



State of Arizona

page



26



D Distributing a needs questionnaire--Biannually, the AOC disseminates a needs questionnaire to the counties asking them to identify any juvenile treatment services that they need, but for which there are none or not enough providers. The responses to the questionnaire help the AOC to develop a solicitation for services that is based on counties' identified needs. The AOC does not actively identify or recruit treatment services vendors to address gaps within its vendor network. According to AOC management, developing an interest among potential vendors to provide treatment services happens informally at the county level. However, according to AOC management, the AOC's solicitation process and Web site, which includes a Web page for vendors interested in contracting with the AOC, represent examples of how the AOC recruits new vendors. Additionally, although the counties do not actively recruit service vendors, if they become aware of a vendor that can provide needed services, they will refer this vendor to the AOC's Web site. D Developing a solicitation for services--According to an AOC manager, once the need for a particular treatment service has been identified for a county, the AOC develops and issues a solicitation for proposals from treatment service vendors. When finalized, the solicitation is posted on the AOC's Web site. Additionally, according to AOC management, the AOC notifies all current and prequalified vendors of the solicitation in writing prior to positing it on the Web site. R Receiving and reviewing proposals--The AOC requires interested independent and full procurement vendors to respond with a treatment services vendor application or proposal that explains the service that the vendor will provide and how they plan to meet the specifications in the solicitation. These proposals are submitted using forms prescribed by the AOC and available on the AOC's Web site. According to AOC management, after receiving a proposal, the AOC and the counties where the services will be provided review the submitted proposal. Specifically, the AOC conducts an administrative evaluation of the proposal, which involves verifying information such as past contract performance, equal access and nondiscrimination, and insurance requirements. Counties conduct a programmatic evaluation of the vendors' proposal, which involves a review of the vendors' qualifications and ability to provide the proposed services. Each county's evaluation includes a scoring process that helps the AOC to determine whether the vendor can meet a county's treatment services needs. A Awarding a contract--According to AOC management, prior to awarding contracts, the AOC negotiates the fee-for-service rate with the provider in an effort to obtain the best rate for the service. Although the AOC attempts to negotiate favorable terms for the State, according to AOC management, rates in rural areas tend to be negotiated at a slightly higher rate because of higher costs the vendor absorbs simply by operating in a rural area. Once these steps are completed, the AOC will award a contract, complete with contract terms and conditions, to vendors that have submitted qualifying proposals.



According to AOC management, recruiting new vendors occurs informally.



Office of the Auditor General

page



27



According to AOC management, the Supreme Court typically enters into 1-year contracts with treatment services vendors, with an option for four 1-year contract extensions, for a total contract term of 5 years. According to AOC management, this is in the best interest of the courts and allows the terms and conditions of the contract to be modified each year based on identified needs or changing circumstances. For example, in its 2007 contracts, the AOC modified its standard terms and conditions to allow for the implementation of the Standardized Program Evaluation Protocol (SPEP). As noted previously, the SPEP is a research-based method for evaluating and improving juvenile treatment programs' effectiveness and compares treatment services programs against best practices shown to reduce recidivism (the rate of re-offense). Programs are scored based on how their characteristics compare with those of model programs. The Supreme Court believes that the SPEP in conjunction with , the risk and needs assessment tools, will provide information necessary for improving the effectiveness of state-funded treatment services programs and will help the juvenile courts effectively refer juveniles to appropriate treatment programs. (See Introduction and Background, pages 3 through 4, for more information on the SPEP) . In addition to the procurement process described above, in some instances, the Supreme Court uses an alternate procurement method referred to as an open and continuous contract process. According to AOC management, this open and continuous contract process is used when the AOC identifies a need for a treatment service that cannot wait until the next contracting procurement period. Vendors that are awarded contracts through the open and continuous process receive nonrenewable contracts that expire at the end of the contract year; however, they may apply for a contract through one of the biannual procurement processes at that time.



AOC should strengthen vendor monitoring

Although the AOC monitors vendors to ensure they comply with contract requirements, it should make some changes to its monitoring schedule and improve its processes for ensuring that vendors correct identified deficiencies. To monitor vendor compliance with contract requirements, the AOC either conducts site visits or requires vendors to conduct and submit self-audits. However, the AOC should improve its monitoring efforts by revising the monitoring schedule to include site visits of some of its smaller vendors and conducting more timely reviews of corrective actions.



AOC monitors vendors' compliance with contracts--The AOC uses the

following two methods to monitor vendors: S Site Visits--AOC conducts monitoring visits for a number of its contracted vendors annually. To determine which vendors will receive a site visit, the AOC selects several specific treatment services categories that it plans to review

State of Arizona

page



28



based on how often they have been provided or if they represent a large portion of the AOC's treatment services expenditures. According to an AOC manager, once it identifies the treatment service categories it plans to review, the AOC then determines how many of its contracted vendors billed at least $20,000 in the treatment service category, and each of these vendors receives a site visit. Vendors that are billed less than $20,000 for a selected treatment service category are generally not selected for a site visit, although according to AOC officials, it may include vendors that fall slightly below that figure.



Areas reviewed during site visits

1. 2. 3. 4. 5. 6. 7. 8. Licensing status Client files Personnel files Interviews with staff, clients, and probation officers Vendor policies and procedures Financial records Environment Vendor's approved contract service proposal



Source: AOG staff analysis of AOC monitoring feedback reports The AOC employs four part-time contract monitors to and information from AOC officials. conduct the site visits. During a monitoring site visit, monitors review eight broad areas for contract compliance. As indicated in the textbox, these areas include reviewing client files for required documentation of the treatment services provided, personnel files for documentation of education and qualification to work with juveniles, financial records to ensure that the Supreme Court has been properly billed for services rendered, and the vendor's policy and procedure manual. Contract monitors also conduct an environmental review to check for health and safety issues, and, according to AOC officials, review the vendor's approved contract service proposal. The contract monitors typically review two to six personnel and client files during the site visit. Finally, the contract monitors interview vendor staff, clients, and probation officers to confirm information obtained during the monitoring review and review applicable reports from licensing agencies.



Following the completion of the site visit, the contract monitors prepare a report documenting the scope and results of their review, including the noncompliance issues identified during the visit. Based on auditors' review of a random sample of 16 site visit reports from fiscal years 2005 and 2006, typical noncompliance issues noted in these reports included missing billing and treatment information from client files, such as service authorization forms and provider notes; incomplete policies and procedures; and personnel files missing some required information, such as documentation of training and supervision. Once the AOC issues the report to the vendor, it requires the vendor to submit a corrective action plan (CAP). In the CAP the vendor should , identify the specific steps that will be taken to correct the identified deficiencies, as well as how it will revise its processes to prevent the identified deficiencies from reoccurring, and include documentation demonstrating the corrective action taken. These CAPs are reviewed by the contract monitors to ensure that the plans will adequately resolve identified noncompliance issues. In addition, contract monitors may conduct follow-up visits to assess vendors' progress in complying with site visit recommendations and contract requirements. However, according to AOC officials, because of time and resource constraints, follow-up visits are generally conducted only for vendors



The AOC requires vendors to submit corrective action plans that identify how noncompliance issues will be addressed.



Office of the Auditor General

page



29



that have experienced significant noncompliance issues, at a probation department's request, or for smaller vendors as a result of the self-audit process. In calendar year 2006, the AOC conducted site visits for 41 vendors. These included one visit to follow up on noncompliance issues from a prior monitoring site visit and four visits to investigate reported incidents. During 2006, four types of services were reviewed, including outpatient services, outof-home care, day programs, and intensive in-home therapeutic interventions. For calendar year 2007, as of June 2007, the AOC had reviewed 20 vendors covering three types of treatment services. These services are foster care, outof-home care, and competency restoration.

The AOC requires vendors that do not receive a site visit to complete and submit a self-audit.



S Self-Audits--Since the AOC does not have the resources to conduct site visits of all of its contracted vendors annually, it requires several of its vendors that have not been selected for a site visit, but that provided services in the treatment categories selected for review, to complete and submit a self-audit. The self-audit consists of the completion of an audit questionnaire developed by the AOC. The audit questionnaire asks the contracted vendor if it complies with specific contract requirements. For example, the questionnaire asks the vendor if staff files contain documentation of qualifications, applicable licenses, fingerprint clearance cards, notarized criminal history affidavits, and annual performance evaluations. In addition, vendors are requested to provide copies of vendor forms that can be reviewed by the contract monitors. These vendor forms include client information sheets, assessments, treatment service plans, and discharge summaries. According to AOC staff, the selfaudit questionnaires can vary slightly, depending on the service offered. For calendar year 2006, the AOC required 19 vendors to complete self-audits. Although no self-audits had been submitted as of September 2007, the AOC plans to require 25 vendors to complete self-audits during fiscal year 2008 that will include verification of forms used through case file submission. As of September 2007, the AOC had yet to identify which specific vendors would be required to complete self-audits. Although the self-audit process verifies that vendor forms comply with contractual requirements, the process does not ensure that providers use the forms as required. For example, in 2006, although a few vendors voluntarily submitted samples of completed forms for the AOC's review, most vendors submitted copies of blank forms. According to AOC officials, the AOC has added a requirement for vendors participating in the self-audit process to provide a sample of case files to the AOC for review. Therefore, the AOC should fully implement its plan by requesting and reviewing a sample of case files with completed vendor forms to ensure contract compliance.



AOC should conduct site visits of a sample of small vendors--

Because of limited staff resources, the AOC generally does not conduct site visits for vendors who have billed less than $20,000 during the previous calendar year for the treatment services categories selected for review. However, one vendor may

State of Arizona

page



30



provide more than one type of treatment service, but the billed amount for each service could be less than $20,000, resulting in that vendor's not receiving a site visit. For example, during fiscal year 2007, one vendor billed the Supreme Court nearly $46,000 for five different treatment services, but only one treatment service exceeded the $20,000 threshold. This treatment service was not included in that year's services selected for monitoring. As a result, this vendor has not received a monitoring site visit in 2007 and also did not receive monitoring site visits in 2005 and 2006. Instead, this vendor submitted self-audits in 2005 and 2006. According to the AOC's 2001 review of the Juvenile Justice Services Division (Division), internal auditors recommended that the Division should consider periodically monitoring smaller vendors, as it may be beneficial to independently validate compliance with certain contract provisions, such as fingerprinting employees and documenting service provision. Therefore, the AOC should modify its vendor sampling approach to annually conduct monitoring site visits for a sample of its smaller vendors.



The AOC should ensure that a sample of its smaller vendors receive site visits.



AOC should ensure that vendors address deficiencies in a timely manner--The AOC should improve its monitoring follow-up processes to help

ensure that vendors address identified deficiencies in a timely manner. Specifically, the AOC sometimes takes months to respond to a submitted CAP and may take , more than a year to resolve all of the issues identified in the site visit monitoring report. Auditors' review of a random sample of 16 monitoring site visit files from fiscal years 2005 and 2006 found that the median number of days between the site visit and the issuance of the site visit monitoring report was 60.5 calendar days. The median number of days between the report and the first CAP submitted by the provider was 26.5 calendar days. Additionally, the median number of days from submission of the first CAP and the AOC's review and approval of the plan was 48.5 calendar days. Auditors also identified some examples where it took the AOC even longer to review submitted CAPs. Specifically, for 5 of the 16 monitoring site visit files reviewed by auditors, the AOC took more than 8 months to review and approve the submitted CAPs. For example, the AOC conducted a site monitoring visit in September 2005, and in November 2005, it sent the vendor the site visit monitoring report, which detailed several noncompliance issues. In December 2005, the vendor responded with a CAP However, the AOC took over 8 months to review this . CAP before denying it, and then took another 5 months to review and deny the vendor's revised CAP In March 2007, nearly 1� years after the September 2005 . site visit, the AOC sent a CAP approval letter. This letter indicated that many of the noncompliance issues noted in the September 2005 site visit had been resolved, and also noted that four noncompliance issues were still not resolved, but would be reviewed during the next scheduled site visit. According to AOC officials, delays in responding to the submitted CAPs are often due to the contract monitors needing to gather additional information required to substantially evaluate the CAP . The AOC has begun drafting policies that will specify the time frames for the review and approval of vendor-submitted CAPs. Therefore, the AOC should finalize, implement, and follow its policies for the timely review and approval of CAPs.

Office of the Auditor General

page



31



Recommendations:

1. The AOC should fully implement its plan to improve its self-audit process by requesting and reviewing a sample of case files with completed vendor forms to ensure vendor contract compliance. The AOC should modify its vendor sampling approach to conduct annual monitoring site visits for a sample of its smaller vendors. The AOC should finalize, implement, and follow its policies for the timely review and approval of vendor corrective action plans.



2.



3.



State of Arizona

page



32



OTHER PERTINENT INFORMATION

During the course of the audit, auditors collected information regarding the Interagency Integration Coordination Initiative, a collaborative state-wide effort to coordinate treatment services between multiple state agencies for dependent and delinquent juveniles and their families. Dually Adjudicated--A child who is found to be dependent or under court jurisdiction pending an adjudication of a dependency petition and who is alleged or found to have committed a delinquent act. Dependent--A child who is adjudicated to be: need of proper and effective parental care and control, but has no parent or guardian willing or capable of exercising such care and control;

In



Interagency Integration Coordination Initiative

In May 2006, various state and county agencies and community advocates merged to create a state team that, in turn, initiated the Interagency Integration Coordination Initiative (Initiative). This Initiative focuses on improving the coordination of services for youth who are involved in both the child welfare and juvenile justice systems. In addition to signing a letter of agreement, involved agencies implemented a Framework for Interagency Practice Protocol and identified several potential outcomes to focus the Initiative's efforts.



or not provided with the necessities of life;

Destitute



in a home that is unfit as a result of abuse, neglect, cruelty, or depravity by the person in custody of the child;

Living



Initiative focused on coordination of juvenile services--In May 2006, multidisciplinary teams from each

Arizona county and the State Team, consisting of representatives from the Governor's Office, Department of Economic Security (DES), Department of Health Services (DHS), Arizona Department of Juvenile Corrections (ADJC), the Office of the Attorney General, the Administrative Office of the Courts (AOC), Regional Behavioral Health Authorities (RBHAs), and community advocates and family representatives participated in a summit meeting to help promote greater integration in the provision of services to children and families. Dually adjudicated youth, or youth who are dependent and delinquent, receive various services from state systems,



age 8 and found to have committed an act that would be adjudicated as delinquent or incorrigible in an older child; and/or

Under



and alleged to have committed a serious offense.

Incompetent



Delinquent--A child who commits a delinquent act, meaning an act that, if committed by an adult, would be a violation of federal law or the law in any state.

Source: A.R.S. ��8-201(10)(11)(13)(a)(i-v) and 8-271(1).



Office of the Auditor General

page



33



including the behavioral health, juvenile justice, and child welfare systems. Based on the work of the State Team at the May 2006 Summit and a review of available information pertaining to Arizona's maltreated and/or justice-involved youth, three possible areas of focus were identified: developing protocols to serve dually adjudicated youth, preventing dependent youth from entering the juvenile justice system, and investigating why Arizona has a high rate of youth who enter the juvenile justice system before entering the child welfare system. According to the Governor's Office for Children, Youth and Families, improved coordination among child-serving systems at both the state and local levels can identify youth and families at risk for multiple systems' involvement earlier, provide more comprehensive and effective services to those families and youth, and cultivate improved outcomes for the youths. To demonstrate support for dually adjudicated youth and their families, and continued support for and commitment to implementing the Interagency Practice Protocol for services to Dually Adjudicated Youth and their Families, participating agency directors and/or their designees signed a letter of agreement in January 2007. Agency officials from the DHS, the DES, the ADJC, and the Committee on Juvenile Courts signed this letter.1 By signing this letter, the agencies agreed to various stipulations, including that: based on interagency collaboration and family involvements are essential to the timely and effective delivery of services to youth and families.

Partnerships



requires a sustained and effective state-wide collaboration between professionals and family members to fully address the complex needs of dually adjudicated youth and their families.

Arizona Effective



collaboration and partnerships will result in:



1.



Better coordinated responses to youth involved in child welfare, behavioral health, juvenile justice systems, and family support agencies; Fewer moves and placements of youth in out-of-home care; Increased number of youth living in home-like and community-based settings geared to ensure the child's success; Reduced frequency of delinquent behavior; Reduced number of younger siblings who enter the juvenile justice system;



2. 3.



4. 5.



1



The Committee on Juvenile Courts (COJC) was established to facilitate communication and problem solving among the juvenile court judges in regard to juvenile court matters. The COJC helps the Committee on the Superior Court and the Arizona Judicial Council develop and implement policies designed to improve the quality of justice, access to courts, and efficiency in court operations. It identifies the needs of the juvenile courts to better serve juveniles facing delinquency and dependency issues within the jurisdiction of the court and for children otherwise involved in the judicial system.



State of Arizona

page



34



6. 7.



Strengthened youth and families; and Safer communities. identification and intervention will:



Early



1.



Reduce the number of children in out-of-home care who are experiencing delinquency problems; Reduce the risk of abuse and neglect to children in families experiencing multiple stressors; and Reduce the number of children who have suffered abuse and neglect and who later become involved in delinquent activities.



2.



3.



education and training will be required to partner agency staff, administrators, youth, and families, as well as the public in general.

Continuous



participating partner agency is committed to developing systemic changes to support the Framework for Interagency Practice Protocol.

Each



participating partner agency is committed to ensuring that all staff/contractors are familiar with and follow this protocol.

Each



Framework for Interagency Practice Protocol--The



Framework for Interagency Practice Protocol (framework) was developed by the Dually Adjudicated Youth Workgroup, consisting of both state and local level representatives from the DES' Administration for Children, Youth, and Families; the AOC; the DHS; the Governor's Office for Children, Youth and Families; County Probation Departments; and community advocates to identify agency responsibilities and advocate the use of best practices to coordinate the provision of services to dually involved youth. The goal of this protocol is that all of Arizona's youth and families with multiple needs will have access to a coordinated set of services, tailored to their needs. The framework includes the following elements: A Agency participation in various coordination activities--Under the protocol, every county has, or will create, a team that includes the youth, parent/guardian, service providers, current caregivers, and agency professionals responsible for providing and monitoring services. Agencies involved may include any combination of the following: Juvenile Probation, the ADJC, the RBHAs, Child Protective Services (CPS), and Adult Probation and Parole. The purpose of this team includes engaging the child and family in identifying their strengths and service needs, developing individualized service plans based on identified needs, engaging the child and family in decision-making, and monitoring the implementation of the individualized plan. Participants in these child and family teams agree to consult with the



Office of the Auditor General

page



35



child and family or other multi-disciplinary team prior to making a recommendation to terminate services, share court actions in a timely manner with the child and family team and with relevant partner agencies, and communicate decisions that occur outside of the planning team to the court and partner agencies. Every county will also have an interagency team that will address systemic or administrative issues. These include barriers to service delivery, including funding and placement issues, systems accountability, system reform, and family involvement across agencies. This team will also develop a training plan for agency staff involved in this Initiative regarding the protocol within the framework, and provide the professional staff and parents needed to assist in delivering the training within agencies and to the public. U Use of best practices encouraged--Involved agencies have also agreed to follow identified best practices to improve the coordination of services for dependent and delinquent youth. For example, the courts can combine delinquency and dependency hearings when both matters are proceeding simultaneously. By combining hearings, it would be easier for the child, parents or guardians, representatives from CPS, Juvenile Probation, and other involved agencies to attend and determine issues and coordinate services. Involved agencies, such as CPS, the ADJC, and/or Juvenile Probation, should then work collaboratively to support the youth's compliance with the courtordered terms in the program in which they have been placed. Another identified best practice is that upon referral, the RBHA should complete a Comprehensive Assessment of the youth, focusing on immediate presenting problems, safety, and collaborative services planning.



Potential outcomes of Initiative--Finally,



the State Team has identified four potential outcomes, and associated strategies and action steps to reach the outcomes. These potential outcomes include: and case management across agencies and the courts that protect the interests of the youth and their families and promote optimal decision-making and case planning.

Information-sharing



of dually involved youths' families are addressed and the number of younger siblings who enter the juvenile justice system is reduced.

Needs Reduce cases of dually involved youth in the child welfare, juvenile justice, and



criminal justice systems. collection system that provides aggregate data for law, policy, and program development, as well as the capacity to measure achievement of system and child outcomes.

Data



State of Arizona

page



36



Bibliography

This bibliography contains literature and guidelines from national juvenile justice organizations used to identify best practices in assessing risk and needs, developing case plans, and monitoring progress. Grossmann, David E. and Maurice Portley. Juvenile Delinquency Guidelines: Improving Court Practice in Juvenile Delinquency Cases. Reno, NV: National Council of Juvenile and Family Court Judges, 2005. National Center for Juvenile Justice. Desktop Guide to Good Juvenile Probation Practice. Pittsburgh, PA: National Center for Juvenile Justice, 2002. U.S. Department of Justice. Guide for Implementing the Comprehensive Strategy for Serious, Violent, and Chronic Juvenile Offenders. Washington, D.C.: U.S. Department of Justice, Office of Justice Programs, Office of Juvenile Justice and Delinquency Prevention, May 1995. Wiebush, Richard G. Graduated Sanctions for Juvenile Offenders: A Program Model and Planning Guide. Reno, NV: National Council of Juvenile and Family Court Judges, Juvenile Sanctions Center, 2003.



Office of the Auditor General

page



37



State of Arizona

page



38



AGENCY RESPONSE



Office of the Auditor General



State of Arizona



SUPERIOR COURT  JUVENILE PROBATION DEPARTMENT Maricopa County

DURANGO FACILITY � 3125 West Durango Phoenix, AZ 85009-6292 � (602) 506-4011 � (602) 506-4143 (TTD) SOUTHEAST FACILITY � 1810 South Lewis Street Mesa, AZ 85210-6234 � (602) 506-2619 � (602) 506-2260 (TTD)



Carol L Boone� Chief Juvenile Probation Officer November 26, 2007 Debbie Davenport, Auditor General Office of the Auditor General 2910 N 44th Street, Suite 410 Phoenix, Arizona 85018 Dear Ms. Davenport, The Executive Management and Treatment Management teams have reviewed the performance audit report, dated November 21, 2007, in reference to Juvenile Treatment Programs in Maricopa County. This performance audit and objective assessment of how we are addressing the needs of juveniles is very timely. The Maricopa County Juvenile Probation Department (MCJPD) is currently updating and developing operating policies and procedures. Your findings and recommendations will be used to improve case management and the delivery and monitoring of treatment services to youth and families. As the department moves forward, MCJPD will actively participate and collaborate with the Administrative Office of the Courts (AOC) and other County Probation Departments to improve juvenile treatment programs and services. The Executive Management Team has targeted process improvement using best practices as a priority in working with juveniles in the system. There are three major areas identified in the performance audit that the department will focus on as a result of your findings and recommendations. These areas are; risk and needs assessment, case planning and monitoring treatment services. I want to take this opportunity to thank you and your team for the insightful information provided and the professional conduct displayed throughout the entire process of this audit. The final report will help the department thoroughly review strengths and areas where we have opportunities for improvement. The MCJPD is looking forward to working with the AOC to update, develop and implement policies and procedures that will help us become a model Juvenile Probation Department. Sincerely, Carol Boone Chief Juvenile Probation Officer



1. Risk Assessment The Administrative Office of the Court (AOC) developed a Risk/Needs tool for all counties in the State of Arizona to utilize, as directed in A.R.S. 8-246. The current assessment tool is not utilized effectively by probation officers. The AOC recently established a workgroup to improve the risk / needs assessment process and make it a functional and easy to use tool that will promote consistent use by probation officers. The Maricopa County Juvenile Probation Department (MCJPD) will continue to work with the AOC and other participating counties to implement a new assessment tool. Policy and Procedure Development MCJPD is working to revise and amend current policies and procedures. The risk assessment process is one such policy being updated. The implementation of such a revised policy will improve the completion rate of required assessments, and to establish timelines for reassessments as an ongoing case management process. Disposition recommendations The current risk/needs assessment does not facilitate matching the right youth with a treatment program. It is the goal AOC needs workgroup to make the revised assessment tool one that is a useful tool to assist in making appropriate disposition recommendations to the Court. Timelines MCJPD will ensure probation staff are educated as to the timeframe required when completing risk and needs assessments. Policy and procedure will dictate assessments shall be completed prior to disposition in order to facilitate using the information when making treatment and supervision recommendations to the Court. 2. Case Plan Development A case plan is a treatment tool that will assist the probation officer in monitoring the progress of treatment. The AOC is committed to creating an assessment tool that will create a treatment plan based on assessment results. This was an original goal of the previous risk / needs assessment, and will be a benefit of a revised tool. Best Practice Information will be taken from the risk / needs assessment to ensure it conforms to Best Practice criterion including the following: clear, specific goals, meaningful objectives, details that outline the process for accomplishing goals and realistic timeframes for completion of identified goals. This expectation will



1



be addressed in policy and procedure revision, which is being updated by MCJPD. Timelines Case plans are to be created within 45 days of a child being placed on probation. This expectation is not new, but will be monitored for compliance in the future. MCJPD believes the development of an assessment tool that incorporates a case plan feature will assist officers in meeting this goal. Review of Case Plans Best Practice has determined that case planning should be a dynamic process, in order to be meaningful. Case plans will be reviewed at regular intervals to ensure youth are working toward goals. Revised policy will require case plans to be used as a living document. 3. Monitoring of Services Documentation of Progress MCJPD will work with the AOC to ensure the documentation of progress toward treatment goals. It is important contracted providers provide progress reports to probation officer that outline the juvenile's movement toward treatment objectives. The probation officer may also conduct reassessments of a juvenile's risk to determine his/her progress towards goals. This information will be kept in the file and shared with the Court as required. MCJPD will establish an internal process to monit