Research and evaluation: building infrastructure and capacity in Arizona

              Arizona Criminal Justice Commission
Statistical Analysis Center Publication
Our mission is to sustain and enhance the coordination, cohesiveness, productivity and effectiveness of the Criminal Justice System in Arizona



Research and Evaluation: Building Infrastucture and Capacity in Arizona



October



2005



ARIZONA CRIMINAL JUSTICE COMMISSION



J.T. McCANN Flagstaff Police Department Chief JOSEPH ARPAIO Maricopa County Sheriff DUANE BELCHER Board of Executive Clemency Chairperson CLARENCE DUPNIK Pima County Sheriff



Chairperson



ROBERT CARTER OLSON Pinal County Attorney



Vice-Chairperson



JIM BOLES City of Winslow Mayor



DAVID K. BYERS Administrative Office of the Courts Director TERRY GODDARD Attorney General



TONY ESTRADA Santa Cruz County Sheriff



BARBARA LAWALL Pima County Attorney



ROD MARQUARDT Mohave County Chief Probation Officer RALPH OGDEN Yuma County Sheriff



TOMMIE CLINE MARTIN Gila County Board of Supervisors



RICHARD MIRANDA Tucson Police Department Chief LINDA SCOTT Former Judge



DORA SCHRIRO Department of Corrections Director ROGER VANDERPOOL Department of Public Safety Director



ANDREW P. THOMAS Maricopa County Attorney



VACANT Police Department Chief



JOHN A. BLACKBURN, JR. Executive Director STEVE BALLANCE MITCH HALFPENNY

Sr. Research Analyst



Statistical Analysis Center Director



JOY LITZENBERGER

Research Analyst



JANICE SIMPSON

Research Analyst



ACKNOWLEDGEMENTS

The Arizona Criminal Justice Commission's Statistical Analysis Center would like to thank all those who contributed to this report. Particular recognition goes to the Governor's Office for Children, Youth and Families for their diligent work on this project. The Statistical Analysis Center would also like to thank the Criminal Justice Research Advisory Group for their participation and support of this project.



SPECIAL THANKS TO:

Jean Ajamie, Arizona Department of Education Bobbie Chinsky, Arizona Administrative Office of the Courts Elizabeth Eells, Maricopa County Juvenile Court Charles Katz, Arizona State University Richard Kennedy, Arizona Administrative Office of the Courts Judy Krysik, Arizona State University Sheila Hoppe, Governor's Office for Children, Youth and Families Denise Muller, Arizona Department of Education Richard Porter, Department of Vital Statistics Nancy Rodriguez, Arizona State University John Vivian, Department of Juvenile Corrections Nicole Yancey, Governor's Office for Children, Youth and Families

This project was made possible by Grant No. AZ29-2004-001 awarded by the U.S. Department of Justice's Office of Juvenile Justice and Delinquency Prevention, Juvenile Justice Evaluation Center. The opinions, findings, and conclusions or recommendations expressed in the publications are those of the author(s) and do not necessarily reflect the views of the Department of Justice. This document is available in alternate formats by contacting the Commission Office at (602) 364-1146.



Table of Contents

Executive Summary Introduction Purpose Methodology i 1 2 3



4 Background Evaluation as a National Priority 4 4 Statistical Analysis Centers 5 Juvenile Justice Evaluation Center 5 Bureau of Justice Assistance Bureau of Justice Assistance � Training and Technical Assistance Program 6 6 Bureau of Justice Assistance � Regional Conference Series 7 National Institute of Justice � Office of Research and Evaluation 8 Performance Assessment Rating Tool Evaluation Efforts in Other States 9 10 Arizona Criminal and Juvenile Justice Infrastructure Administrative Office of the Courts (AOC) � JOLTS 11 Arizona Youth Survey (AYS) 12 Uniform Crime Reports (UCR) 12 National Incident Based Reporting System (NIBRS) 13 Criminal History Records Central Repository (CHR) 14 Social Indicator 15 United States Census 15 Youth Risk Behavior Surveillance System (YRBSS) 16 Department of Juvenile Corrections (ADJC) 16 Criminal Justice Research Advisory Group (CJRAG) CJRAG Infrastructure Review Future Plans Statewide Efforts Arizona Evaluation Network (AZENet) State Epidemiological Workgroup Community Health Analysis Areas (CHAAs) IMAP Center for Violence Prevention and Community Safety Re-Entry Council 18 18 22 24 24 25 26 28 31 32



Survey Interviews of JABG, Weed and Seed Sites, and Drug Free Community Coalitions JABG Subgrantees Drug Free Community Coalitions Weed and Seed Findings Data strengths Data Limitations Inter agency support and collaboration Infrastructure Recommendations Developing Awareness for Evaluation Increasing Data Sharing and Accessibility Expanding the Data Infrastructure Conclusion Bibliography Appendix A



34 34 35 37 39 39 39 40 40 41 41 41 41 43 46 48



Executive Summary

Funding sources for the juvenile justice system have followed a nationwide trend of requiring proof of program success to maintain financial support. Evaluating programs for efficiency and effectiveness has become more important as sources of funding become increasingly scarce. This leaves many programs in a difficult position of needing to show efficiency and progress, without sufficient resources or knowledge to set up formal program evaluations. This report seeks to look at the state of Arizona's evaluation capacity as a whole, its ability at the program level to utilize available resources, and report on needed improvements to the system to allow for program evaluation. Reliable database structures are tremendous assets to decision makers and program evaluators. In an environment where funding increasingly depends on results, it is necessary to ensure that accurate and complete data is collected and made available. Data collection should be a key component of any program design. Statewide data collection standards should be implemented if information is to be accessible and beneficial to the criminal justice and juvenile justice systems. Establishing and building an atmosphere conducive for research and evaluation within the criminal and juvenile justice systems is necessary for demonstrating effectiveness. More and more, proving results and effectiveness are a requirement for funding programs and/or the continuation of existing programs. This project consisted of a number of strategies to improve access to currently available resources, and identify the strengths and weaknesses of the system. The first phase involved bringing together researchers and evaluators from throughout the state to strengthen communication and determine common issues facing these experts in the field. This group was named the Criminal Justice Research Advisory Group. These meetings identified a number of areas that could be strengthened to increase the state's evaluation capacity, and members worked together on a number of statewide projects to lend their expertise. After seeing the benefits of working together to strengthen partnerships and share resources and expertise, the group recommended that it become a regularly meeting body. In addition, the Statistical Analysis Center sought feedback from coalitions and subgrantees on issues pertaining to evaluation, data collection and general challenges relative to demonstrating effectiveness. For this purpose, the staff of the Statistical Analysis Center at the Arizona Criminal Justice Commission surveyed three distinct groups of programs covering multiple areas within the criminal and juvenile justice system: Weed and Seed projects, Drug Free Communities, and Juvenile Accountability Block Grant subgrantees. The feedback review showed a general willingness to provide information, but a lack of understanding as to what information would demonstrate the effectiveness of

Research and Evaluation: Building Infrastructure and Capacity in Arizona



i



programs. Examples of effective data would be: baseline data from which to measure subsequent reports; measures designed to indicate real progress; corroborative information that provides context for a program; inclusion of outside issues that affect reported measures, and reporting successes that are not included in the measures provided. It should be noted that many of the subgrantees did not understand that this information was needed or useful. Suggestions for improvements included more training on how to report on the use of funds, requiring evaluations of programs, and ongoing review of submitted reports for completeness and usefulness, with feedback as necessary. It is important to note that primary focus of subgrantees is assuring that mandated reporting requirements by the funding agency are met and may not be directed toward developing measures that speak to the overall performance of the program. Drug Free Communities coalitions most frequently relied on Arizona Youth Survey data for reporting efforts, as well as internal record keeping. During this project, data sets specific to some of the Drug Free Communities were created and presented to the projects. Many indicated that prior to this data being provided, they had been unable to provide data in reports. Weed and Seed sites were the most likely to express a lack of ability to get needed information, especially drug information at a level low enough to be useful. They indicated that juvenile arrest data was much needed for their projects, especially as it pertained to drug offenses. Juvenile Accountability Block Grant subgrantees were the most likely to report that they had all needed information. These programs primarily relied on JOLTS data and other internal records. Types of data identified as needed by these programs included more access to JOLTS data, AYS data at the city level, gun and drug arrest data for juveniles, and recidivism data for juveniles. These data sources were similar to the ones identified by the Criminal Justice Research Advisory Group. While there appeared to be statewide recognition of the need to track progress and show effectiveness to those outside the immediate community, funded entities and the CJRAG group showed areas where the information infrastructure in Arizona needs to be improved. Some important existing resources within the state were highlighted as examples of the type of infrastructure that must be improved. Most noted resources included JOLTS, Uniform Crime Reports, and the Arizona Youth Survey. Other sources included NIBRS, criminal history records, Social Indicator data and U.S. Census data. With all of the resources highlighted, programs pointed out the need for data at lower levels for better analysis. Evaluation of programs has always been an important process for determining whether a project is expending resources and meeting goals in an effective manner. It also provides an opportunity to reassess whether the original intent of the program's mission and design are being followed. However, program evaluation can be daunting for those not trained in either developing a program in a manner that can be evaluated or in



Research and Evaluation: Building Infrastructure and Capacity in Arizona



ii



methods of evaluation. In the past, this along with a lack of adequate funding dedicated for research and evaluation has precluded programs from being assessed or evaluated. It is important for Arizona to become a leader both in showing the effectiveness of programs, and modeling new programs on proven methods. The most basic elements of this goal are to provide a stronger infrastructure for assessing programs and policies within the criminal and juvenile justice systems and to increase priority of research and evaluation within the state. The improved infrastructure could begin by taking incremental steps for integrating data systems to allow for improved communication with each other and make them accessible to researchers and evaluators. Decision makers are encouraged to utilize this report, Research and Evaluation: Building Infrastructure and Capacity in Arizona, in developing strategies for improving the criminal justice data infrastructure and for assuring that adequate resources are dedicated toward research and evaluation in Arizona. Only through such an approach can policy makers be provided information to develop policies that have the greatest impact in the areas that are most in need of additional resources.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



iii



Introduction

Arizona's current criminal and juvenile justice systems face significant pressure. Over the past 10 years, Arizona has had a considerably higher reported crime rate than the rest of the country (Crime Trends, 2005) and was number one in overall Part I crime in 2003 (Crime in the United States, 2004). Arizona's exploding population, increased homeland security demands and dwindling federal and state resources have directly contributed to the challenges confronting the Arizona criminal and juvenile justice systems. In spite of these hurdles, Arizona's criminal justice policymakers are committed to addressing challenges by directing limited resources toward innovative and reliable crime prevention strategies in the state. To do this, a foundation supporting the effective and ongoing evaluation of programs and systems will be critical. Evaluating programs for efficiency and effectiveness has become more important as sources of funding become increasingly scarce. On both a state and national level, agencies and programs are being required to justify their existence. This trend of program justification intensifies the need for evaluation to be ongoing and systematic. Establishing and building the evaluation capacity into criminal and juvenile justice programs and services is a logical response to the concept of demonstrated effectiveness as a requirement for program funding or continuation. Unreliable, incomplete, or inaccessible data structures can be major stumbling blocks for decision makers, no matter how willing they may be to cooperate with and encourage program evaluation. In an environment where funding increasingly depends on the results of a program evaluation, it is necessary to ensure that accurate and complete data is collected and made available. As new programs are designed, data collection should be a key component. Statewide data collection standards should be implemented for information to be accessible and beneficial to the criminal and juvenile justice systems. This process of making data more accessible will likely result in the identification of information gaps and barriers to program performance. At the same time, enhanced infrastructure for ongoing evaluation provides a basis to identify both program obstacles and successes. Improvements to state and local data collection tools and data reporting methods will build the bridge toward recognizing effective approaches that can be replicated or tailored to address specific criminal justice needs or issues. In spite of the increasing challenges, Arizona has been recognized as having one of the leading juvenile justice systems in the nation. The Deloitte study was commissioned by the state legislature to evaluate the primary components of the Arizona juvenile justice system. The report noted that "It is our assessment that the Arizona juvenile justice system is one of the best in the nation" (Deloitte, 2001). While the state can be proud of its accomplishments, ongoing improvement of the juvenile justice system has also been defined as a priority. The Juvenile Online Tracking System (JOLTS) was

Research and Evaluation: Building Infrastructure and Capacity in Arizona



1



implemented statewide in the 1990's and provides a wealth of information regarding youth within the juvenile justice system. With support from the Justice Research and Statistics Association's (JRSA) Juvenile Justice Evaluation Capacity Building Project, the Arizona Criminal Justice Commission and its research component, the Statistical Analysis Center (SAC), formed a working partnership with the Governor's Juvenile Justice Commission. This partnership sought to develop approaches for building the capacity for conducting evaluations in Arizona. The goal of the project is to advance efforts to improve data collection and dissemination practices on the state and local level to ensure that reliable data is available to inform program practice and provide for data-driven decision making. By forming partnerships with Governor's Office of Children, Youth, and Families (GOCYF), Administrative Office of the Courts (AOC), and others within the juvenile justice system, it was hoped that this project would spur improvements in the quality and quantity of data collected, as well as build an environment where evaluation is considered a base component of criminal and juvenile justice programs. The presentation of the report is contained in the following sections: 1) Purpose 2) Methodology 3) Background 4) State Projects 5) Survey Results 6) Findings 7) Recommendations 8) Conclusion. The partnership directed the focus of its research to more profoundly understand how these elements could produce a vital impact on program monitoring, performance measurement and impact evaluation. A large part of developing an evaluation plan included bringing together the evaluators and data holders throughout the criminal and juvenile justice system to build partnerships and find ways to improve current efforts for information sharing for the purpose of evaluation.



Purpose

The purpose of this project was to advance efforts in developing a comprehensive statewide juvenile justice evaluation plan. For this purpose, the Statistical Analysis Center developed a plan for assessing the availability of data relative to conducting evaluations by programs and coalitions in Arizona. This plan included surveying subgrantees to identify capacity building needs for data collection. The Statistical Analysis Center worked with the Governor's Office for Children, Youth and Families (GOCYF) and the Administrative Office of the Courts (AOC) to identify current data collection mechanisms and build a foundation for sharing standardized data. In addition, this project sought to increase awareness among policy makers in juvenile justice programs regarding the need to evaluate their programs and their ability to assess information necessary for this purpose. Related to this effort, the Statistical Analysis Center sought opportunities to assist other agencies, particularly in the area of data collection and program evaluation when possible.

Research and Evaluation: Building Infrastructure and Capacity in Arizona



2



Initially, the main purpose of this project was to investigate some of the challenges to conducting evaluations of juvenile justice programs in Arizona. During this review CJRAG found that many of the issues facing the juvenile justice system overlapped with issues facing the criminal justice system as a whole. For that reason, some of the group's focus was on criminal justice, as well as juvenile justice, issues.



Methodology

Throughout the planning stages of this project, it was understood that in order to thoroughly analyze the issues surrounding program evaluation, a diversified approach was necessary. Processes were set in motion to explore the juvenile justice system and the evaluation capabilities that included the availability of data, the process, the practitioners and the characteristics of the system itself. The first step of this process involved forming the Criminal Justice Research Advisory Group (CJRAG). This group of juvenile justice research professionals was developed in order to discover the areas that need improvement within the current evaluation system. As recommendations and information came out of this group, other key partners were involved in the project and specific goals identified. The first goal was to identify the current infrastructure within the state for sharing data, evaluating programs, and communicating within the juvenile and criminal justice systems. For this purpose, ongoing meetings with CJRAG both as a group and with individual members provided a basic understanding of this infrastructure. Next, the Statistical Analysis Center developed and administered a phone survey of Juvenile Accountability Block Grant subgrantees, Weed and Seed coalitions and Drug Free Community coalitions. This broad group was chosen in order to get a statewide picture of challenges that programs faced when evaluating their effectiveness. This process had two purposes. The first was an information-gathering function. The second was to introduce resources such as the Statistical Analysis Center (SAC) that are available to these groups, thereby increasing their ability to evaluate effectiveness.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



3



Background

Evaluation as a National Priority Nationally, a stronger emphasis on demonstrating the effectiveness of grant funded programs has caused entities to set research and evaluation as a priority. This emphasis has grown out of efforts to link the results of program evaluations to funding. Many entities at a national level either conduct evaluations of programs, or provide resources to programs to assist in evaluation efforts. These efforts range from the individual Statistical Analysis Centers in each state that conduct research and evaluation at a state level, to the Juvenile Justice Evaluation Center which provides resources to enable entities to conduct program evaluations. The Justice Research and Statistics Association, the National Institute for Justice, and the Bureau of Justice Assistance are all national entities that are focused on improving evaluation at the local level. Some of the efforts put forth by programs such as the Bureau of Justice Assistance require evaluation for continued funding. Other projects such as the Performance Assessment Rating Tool (PART) are used by policy makers to assess the success of programs in relation to funding. Each of these agencies shares a common focus in increasing the ability of programs to demonstrate effectiveness which will lead to the more efficient use of limited funds. Although not intended to be all inclusive, the following section provides a brief overview of the agencies noted above. Statistical Analysis Centers Almost every state has a Statistical Analysis Center (SAC). These centers conduct research into topics that directly impact criminal and juvenile justice issues in their state. These research agencies provide a resource to criminal justice related entities by providing data and technical assistance and can often assist programs in conducting research and evaluation, or provide data that will assist entities in conducting research and evaluation. In Arizona, the Statistical Analysis Center operates as the "research arm" of the Arizona Criminal Justice Commission. This center conducts a biennial Arizona Youth Survey, gang surveys, a web based interactive map project called IMAP that includes UCR, AYS and U.S. Census data and studies on issues facing the criminal justice system in Arizona. This center provides data to entities statewide to assist programs with their data needs. Reports produced by this center, along with the IMAP interface can be accessed at http://www.azcjc.gov/sac/index.asp. Nationally, a coalition called the Justice Research and Statistics Association pulls together the various SACs allowing for cooperation, data sharing and training. Links to every SAC, along with publications, can be found at http://www.jrsa.org/.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



4



Juvenile Justice Evaluation Center The Juvenile Justice Evaluation Center is a joint project between the Justice Research and Statistics Association and the Office of Juvenile Justice and Delinquency Prevention. The purpose of this center is to improve the evaluation of state juvenile justice programs. The center is using several methods to accomplish this goal including publishing a number of reports that give guidance to programs attempting to conduct, or prepare to conduct an evaluation. The Juvenile Justice Evaluation Center Online component of this project was developed to provide instant access to online evaluation training and resources. Included on the web site (http://www.jrsa.org/jjec/about/index.html) is an online tutorial designed to inform users about the basics of evaluation. Another facet of the program is funding statewide juvenile evaluation projects conducted by Statistical Analysis Centers. This report is part of that project. The Juvenile Justice Evaluation Center seeks to improve the ability of programs to conduct evaluations and enhance the knowledge base of personnel on how to conduct evaluations. This project was designed to enhance the overall ability of states to conduct evaluations, rather than fund a small number of projects that would have limited utility. By creating models for evaluation and assisting states in understanding the need for, and how to conduct evaluations, this project sought to provide a valuable resource to all programs in need of demonstrating effectiveness in a funding environment that increasingly requires it. The online tutorial, as well as other publications put out by this center are valuable resources for juvenile justice entities. Bureau of Justice Assistance The Bureau of Justice Assistance is a federal agency that provides leadership and services to support local, state and tribal justice strategies. It accomplishes this goal through grant administration and criminal justice policy development. Overall goals of BJA are to (1) reduce and prevent crime, violence, and drug abuse and (2) improve the functioning of the criminal justice system. As part of working to improve the functioning of the criminal justice system BJA works to assist criminal justice agencies in evaluating the effectiveness of funded programs and disseminating program results. Many federal justice grants are administered by the Bureau of Justice Assistance. These grants require that funded programs demonstrate effectiveness. The BJA provides direction and assistance to funded programs through training and one-on-one assistance in evaluating the effectiveness of these programs. The Bureau of Justice Assistance works with criminal justice agencies to enhance coordination at the federal, state and local level, which provides resources beyond grant funding.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



5



Bureau of Justice Assistance - Training and Technical Assistance Program The Bureau of Justice Assistance administers a Training and Technical Assistance program to assist criminal justice agencies with developing and implementing comprehensive, system-wide strategies for improving criminal justice systems and public safety. Included in this mission is providing training and technical assistance to programs in the areas of problem solving, information sharing, curriculum development and publication drafting and dissemination. This program provides a number of services that can assist programs in evaluation including:

� � � � � � � � � � �



Problem assessment. Strategic planning/strategy development. Program sustainability. Cutting-edge practice(s) documentation and identification. Evidence-based practice(s) identification and development. Team building and collaboration. Community partnership development. Staff development. Resource identification and management. Information management. Program evaluation. (http://www.ojp.usdoj.gov/BJA/tta/index.html)



The Training and Technical Assistance Program offers a number of training opportunities to eligible entities which includes any state, tribal, or local agency, or organization involved in activities related to fighting crime or ensuring public safety. Assistance is not limited to areas that the Bureau of Justice Assistance funds, as this program also works to pair programs up with other available resources as needed. More information regarding this program and resources offered can be found at http://www.ojp.usdoj.gov/BJA/tta/index.html including information on how to apply for assistance and a list of training currently offered. Bureau of Justice Assistance - Regional Conference Series Every year the Bureau of Justice Assistance conducts a Regional Conference Series every year to bring together practitioners, policy makers, and BJA leadership to provide training regarding critical issues in criminal justice. The 2005 series was designed to focus on demonstrating and communicating the value of justice initiatives. Focusing specifically on the fact that funding entities' budgets have become tighter recently, this conference is designed to provide training in how to demonstrate the value of criminal justice programs. These conferences are free of charge. For more information, including agendas and registration information, go to http://www.ncja.org/bjaregionalmeeting.html.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



6



National Institute of Justice - Office of Research and Evaluation The National Institute of Justice (NIJ) is the research, development, and evaluation agency of the U.S. Department of Justice. In 2003 NIJ was reorganized, and one of the intentions of that reorganization was to "emphasize outcome and cost-benefit evaluations of criminal justice programs and technologies." There is an emphasis on serving and evaluating programs at the state and local level. NIJ has seven strategic goals: 1. Partner with State and local practitioners and policymakers to identify social science research and technology needs. 2. Create scientific, relevant, and reliable knowledge � with a particular emphasis on terrorism, violent crime, drugs and crime, cost-effectiveness, and communitybased efforts � to enhance the administration of justice and public safety. 3. Develop affordable and effective tools and technologies to enhance the administration of justice and public safety. 4. Disseminate relevant knowledge and information to practitioners and policy makers in an understandable, timely, and concise manner. 5. Act as an honest broker to identify the information, tools, and technologies that respond to the needs of stakeholders. 6. Practice fairness and openness in the research and development process. 7. Ensure professionalism, excellence, accountability, cost-effectiveness, and integrity in the management and conduct of NIJ activities and programs. (http://www.ojp.usdoj.gov/nij/about.htm) NIJ's Office of Research and Evaluation identifies priority issues and then develops, conducts, directs, and supervises research and evaluation activities in accordance with those issues. The research and evaluation can be done by either NIJ staff or by contract with outside researchers. One of the priority issues is Program Evaluation which has three goals: 1. Develop the capacity to conduct cost-effectiveness evaluations of criminal justice programs and technologies. 2. Improve the quality and cost-effectiveness of NIJ evaluations. 3. Improve the utility of evaluation results for policy, practice, and program development in part through interim reporting and timely reporting of final results. Within the Office of Research and Evaluation is the Evaluation Division which oversees NIJ evaluations of other agencies' programs and develops policies and procedures that establish standards for assuring quality and utility of evaluations. NIJ's website (http://www.ojp.usdoj.gov/nij/about.htm) contains links to more than five hundred research studies it has been involved with since 1994, as well its annual reports and journal.

Research and Evaluation: Building Infrastructure and Capacity in Arizona



7



Performance Assessment Rating Tool (PART) The Office of Management and Budget, which provides the data used by the President to propose a budget, evaluates the effectiveness of each program in order to provide this information. Programs which have conducted evaluations on subcomponents of the program are in a better position to show effectiveness. For this reason, Arizona is in a position where it must demonstrate that federally funded projects are effective, or risk losing much needed funding. In order to develop a system by which to gauge the effectiveness of federal programs and grants, the Office of Management and Budget created the Performance Assessment Rating Tool, also known as PART. PART uses four basic measures to assess effectiveness: program purpose and design; strategic planning; program management; and program results. The finalized tool used for the evaluation during FY2004 consisted of over 50 questions, some general and some geared toward a program type. Based on the results of the responses, programs were given one of four overall ratings: 1. 2. 3. 4. "Effective" "Moderately effective" "Adequate" "Ineffective"



A fifth rating, "results not demonstrated," was given if OMB decided that a program's performance information, performance measures or both were insufficient or inadequate (Performance Budgeting: Observations on the Use of OMB's Program Assessment Rating Tool for the Fiscal Year 2004 Budget, p. 11). Those program ratings showed how well each program could demonstrate that they were being effective. An effective program that did not have the documentation to prove effectiveness would be rated poorly.

Section Program Purpose and Design Strategic Planning Program Management Program Results Performance Assessment Rating Tool Description To assess whether � The purpose is clear � The program design makes sense Has the agency set valid programmatic � Annual goals � Long-term goals To rate agency management of program including � Financial oversight � Program improvement efforts To rate program performance on goals reviewed in � The strategic planning session � Other evaluations Weight 20% 10% 20% 50%



Once this review was completed, a recommendation was given regarding funding for the next fiscal year. This emphasis on proving effectiveness trickles down from the

Research and Evaluation: Building Infrastructure and Capacity in Arizona



8



federal program level to the state and local levels. Although it is clear the PART project can be used as part of a political agenda, it is important to note the increased emphasis on assessing and evaluating grant funded programs. Evaluation Efforts in Other States Arizona is not the only state facing the challenge of raising the priority for research and evaluation in their criminal and juvenile justice systems. Administrators and policymakers in other states have identified the need to develop a state evaluation strategy that consists of more comprehensive, standardized, and useful evaluation policies and practices. Some have successfully focused on specific issues: Maine has developed a standardized grant application process; Maryland has developed a model for evaluating Teen Courts; Illinois created a process for forming Juvenile Justice Councils which are responsible for local needs assessments and policy recommendations; and in Iowa the Criminal and Juvenile Justice Planning and Statistical Analysis Center has collaborated with other state agencies to help communities develop sustainable outcome monitoring processes for their programs. In Iowa, Washington, and Pennsylvania efforts to build evaluation capacity took on broader focus. The Iowa Collaboration for Youth Development (ICYD) was created to better align state policies and programs, and to facilitate the use of effective youth development practices in communities throughout the state. The effort is led by the Criminal and Juvenile Justice Planning and Statistical Analysis Center, and more than a dozen state agencies, youth organizations and research institutions have joined the effort. In Washington and Pennsylvania the State Advisory Groups (SAGs) have adopted two very different evaluation strategies to assess the effectiveness of programs and initiatives funded by the Office of Juvenile Justice and Delinquency Prevention's Formula Grants Program. In Washington every program, regardless of size, must be evaluated by an external evaluator (http://www.jrsa.org/jjec/about/wa-pa-2003.html, pg. 3). In Pennsylvania the SAG, which administers the juvenile justice grant awards, has developed a state planning process to strategically determine which programs merit a formal evaluation study. Projects that are considered for evaluation include those that are particularly innovative, controversial, or potentially dramatic in impact; those whose operations and management could benefit by constructive evaluation; and those representing a substantial investment of funds (http://www.jrsa.org/jjec/about/wa-pa2003.html, pg. 3). JRSA conducted case studies of the practices in Washington and Pennsylvania and discovered what they considered to be critical factors in their approach to evaluation. � Juvenile justice specialists in the SAG, who were experienced and on a managerial level, provided consistency and authority in the implementation of an



Research and Evaluation: Building Infrastructure and Capacity in Arizona



9



evaluation strategy. � SAG members who are knowledgeable about research and evaluation. As informed consumers of the evaluation studies, experienced researchers and evaluators are better able to use objective research as part of the planning process and to prioritize program areas that should be given funding. Support for research and evaluation from the SAG, state staff, and state leadership. If formal state evaluation policies and practices are to be implemented, all levels of the state infrastructure that influence the administration of juvenile justice grant awards must agree that evaluation is important. A planned financial commitment to research and evaluation. Washington requires each program to earmark funds for evaluation; Pennsylvania has a separate research and evaluation budget. Strong relationships with the research community. Many states lack the resources to do all evaluations in-house. They both foster relationships with and draw on the expertise of other state agencies, universities, smaller research firms, and independent consultants involved in justice research (http://www.jrsa.org/jjec/about/wa-pa-2003.html, pg. 22).



�



�



�



These efforts to improve evaluation throughout the nation provide valuable insights into what various states have needed and how they have attacked their problems. They should not be viewed as templates for Arizona, but rather as possible approaches to addressing similar challenges in Arizona. Although there are notable differences between Arizona and these states such as size, geographic organization of counties, population growth, and proximity to a border, the solutions derived from others might provide valuable insights for Arizona. Arizona Criminal and Juvenile Justice Infrastructure Researchers and evaluators are dependent on timely and reliable data when conducting evaluations. The Statistical Analysis Center developed a project to assess the current data infrastructure in Arizona, make recommendations on how to improve the current infrastructure, and to improve communication between researchers in the criminal and juvenile justice fields. This project was essentially built around the idea of integrating current data to form a network of available, accurate and current data to be used within the juvenile justice system. In addition to working to strengthen these partnerships, this project proposed to find weaknesses within the current system in order to develop strategies to improve the data infrastructure statewide.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



10



Within the state of Arizona, there are several statewide data collection tools that are accessible in some form that support evaluation efforts. However, much of this data is only available at the county or state level and is often difficult to obtain in a timely fashion. Eight of the main data sources available for evaluating criminal and juvenile justice programs are the Juvenile Online Tracking System (JOLTS), the Arizona Youth Survey, Uniform Crime Reports, Criminal History Records, U.S. Census data, Social Indicator Data, Youth Risk Behavior Surveillance System and the Department of Juvenile Corrections. Although certainly not the only sources of data available or used in evaluations, these eight data sources do represent some of the largest data systems within the juvenile justice system. The following section provides a brief summary and notes some of the limitations in regards to the current structure or accessibility of the data sources.



Administrative Office of the Courts (AOC) � JOLTS

The Juvenile On-Line Tracking System (JOLTS) has been operational in the Maricopa County Juvenile Court since 1979. As of 1993, JOLTS had been implemented in every county, creating a statewide repository for juvenile data. It is currently installed in every juvenile court and detention center in the state. Over time the JOLTS system has moved from being a county level tracking system to a state level tracking system, and has been enhanced considerably since its inception. The JOLTS system is comprised of three separate, but compatible, systems. One system covers Maricopa County, another Pima County, and the third the remaining thirteen counties. Enhancements have been designed to ensure compliance with legislative changes, track new probation department programs, and improve case management. At a JOLTS Steering Committee meeting in October 1993, former Chief Justice Thomas Zlaket spoke of JOLTS saying, "This is our first major success.... It is a model for the whole country. It works and it works well." The Juvenile On-Line Tracking System has approximately 2,600 users across the state. Recent improvements to JOLTS increased interfaces with other systems/applications through the use of MQ (messaging) technology and implemented a new extensive needs assessment. Future plans include automating Terms and Conditions of Probation and case plans (http://www.supreme.state.az.us/jjsd/jolts/Default.htm). Juvenile Accountability Block Grant subgrantees cited the JOLTS program as one of the primary sources they use to obtain information for evaluation purposes. This program collects the information used in the annual Juveniles Process in the Arizona Court System each year. This publication provides a wealth of information at the aggregate level on juvenile justice trends and statistics. The Arizona Administrative Office of the Courts (AOC) is an important partner in future efforts for building the capacity for conducting research and evaluation within the juvenile justice system. Through the collaboration of the AOC and the participation of individual courts throughout the state, the Juvenile Online Tracking System (JOLTS) has



Research and Evaluation: Building Infrastructure and Capacity in Arizona



11



become a statewide system that has increased value for the purpose of evaluating programs and juveniles within the juvenile justice system.



Arizona Youth Survey (AYS)

The Arizona Youth Survey is administered to a statewide sample of youth in Arizona to give detailed information regarding substance use in compliance with Arizona Revised Statute �41-2416. Since 1988, the survey has been administered every two years with significant changes being introduced in the 2002 survey. The AYS uses the risk and protective factor framework developed in 1989 by J. David Hawkins, Ph.D., and Richard F. Catalano, Ph.D., at the University of Washington. The survey provides data and information at state and county levels regarding risk and protective factors, as well as information about drug use, school safety, gang involvement, delinquency and other issues. In the 2004 administration, 266 schools in 15 counties were surveyed resulting in a participation of 40,960 students in the 8th, 10th, and 12th grades across Arizona. From this, data reports were created for individual schools, districts, counties, and the state as a whole. Data from the 2004 administration is being used for the first time to develop municipal level reports. The individual school reports and data are not published but are made available for each participating school. The next survey will be administered in January 2006 to schools across Arizona. If schools are interested in participating in the survey, contact information is available on the ACJC web site at http://azcjc.gov/sac/AYS.asp. In that the survey is conducted every two years, the data is current and comparable to previous results. However, data is needed at lower levels than county and state reports. The municipal reports are valuable and need to be continued. Ongoing efforts are directed at providing state and local decision makers with data at lower levels of geography.



Uniform Crime Reports (UCR)

The Uniform Crime Reports are a common indicator of crime which is reported by law enforcement agencies throughout the United States to the Federal Bureau of Investigations (FBI). "The FBI's primary objective is to generate a reliable set of crime statistics for use in law enforcement administration, operation, and management" (http://www.fbi.gov/ucr/ucr.htm). To ensure consistency in reporting, the FBI provides participating law enforcement agencies with a detailed outline to follow when reporting crime. The outline includes specific definitions of the crimes reported. Seven main classifications of crime were chosen to gauge fluctuations in the overall volume and rate of crime. These seven classifications eventually became known as the Crime Index which includes: Criminal Homicide, Forcible Rape, Robbery, Aggravated Assault, Burglary, Larceny-theft, and Motor Vehicle Theft.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



12



Generalizations based solely on UCR data need to consider limitations inherent to the UCR program. Limitations include the fact that until the passage of Arizona Revised Statute �41-1750 in 1992, participation by law enforcement agencies in UCR data collection was voluntary. Moreover, UCR does not provide details pertaining to the crime such as victim related information, when the crime occurred and where the crime occurred. In addition, when several crimes occur in a single event, only the most serious is reported for UCR purposes. In Arizona, crimes reported to Native American law enforcement agencies are not reflected in UCR statistics. Because of these and other limitations, the number of UCR offenses reported will always be less than what is actually occurring. Uniform Crime Reports provide information both for the adult and juvenile justice system. Uniform Crime Report data is available to researchers and evaluators; however, it is not available below the agency level of data collection. Further accessibility of this data at a lower level of analysis, including geographic information, would greatly assist researchers and policy makers in Arizona.



National Incident Based Reporting System (NIBRS)

The National Incident Based Reporting System (NIBRS) is a national project that will enhance the current UCR program by providing more detailed information than was previously collected. Over the past five years, NIBRS reporting has more than doubled with the number of state certified programs increasing to 18. The Arizona Department of Public Safety is working toward implementing a statewide Arizona Incident Based Reporting System (AIBRS) repository. AIBRS is designed to collect data on each single crime occurrence and on each incident and arrest within that occurrence. One of the significant differences between AIBRS and the traditional UCR summary system is the degree of detail in reporting. The UCR summary system collects information on only eight Part I crimes, known as the Crime Index (murder, forcible rape, robbery, aggravated assault, burglary, larceny-theft, motor vehicle theft, and arson). AIBRS collects information on 22 crime categories made up of 46 specific crime offenses in Group A (arson, assault, bribery, burglary/breaking and entering, counterfeiting/forgery, destruction damage/vandalism of property, drug/narcotics, embezzlement, extortion/blackmail, fraud, gambling, homicide, kidnapping/abduction, larceny/theft, pornography/obscene material, prostitution, robbery, forcible/nonforcible sex offenses, stolen property, and weapon law violations) and 11 Group B offenses (bad checks, curfew loitering/vagrancy violations, disorderly conduct, driving under the influence, drunkenness, nonviolent family offenses, liquor law violations, peeping tom, runaway, trespass of real property, and all other offenses). In Group B offenses, only arrest data is reported. AIBRS information will be forwarded to the Federal Bureau of Investigation and added to the National Incident Based Reporting System (NIBRS) database. (http://www.ojp.usdoj.gov/bjs/nibrs.htm).



Research and Evaluation: Building Infrastructure and Capacity in Arizona



13



AIBRS will provide increased reporting capabilities and provide victim and offender related data and incident specific information. Although AIBRS generally provides more detailed information than is currently available from the Uniform Crime Reports, it has very limited participation in Arizona. At present, there are only five participating law enforcement agencies in Arizona in the AIBRS project, with another scheduled to begin testing soon. Three of those agencies, Yavapai County Sheriff's Office, Apache Junction Police Department, and Springerville Police Department, have received official certification as of the writing of this report. Phoenix Police Department and Yuma County Sheriff's Office are in the testing phase. In addition, AIBRS does not provide information as to the time and location of the event and provides information at the agency level.



Criminal History Records Central Repository (CHR)

The coordination and maintenance of accurate criminal history records is essential to an effective criminal justice system. Quality information provided through an efficient record keeping system is a necessary component not only for individual case processing, but for the success of crime control strategies. Records are reported and accessed by local, state, and federal justice systems. Though most crimes are prosecuted under state law, the individual systems are interrelated and all rely upon criminal justice records. Currently, criminal history records are collected and maintained by each state in a central state repository database. In Arizona, pursuant to A.R.S. �41-2205, the coordination and maintenance of criminal history records in a central repository is fulfilled by the Arizona Department of Public Safety (DPS) which oversees database operations. The repository database, Arizona Computerized Criminal History (ACCH), holds all state offender and criminal history records including fingerprint files, arrest information, and disposition information. Arizona has made great strides in the past decade in maintaining and improving records in the criminal justice system. It is increasingly important to maintain reliable information as the state and its contributing agencies increase their levels of record automation. Record integrity and accessibility are crucial as the direction of the criminal justice system progresses toward a more integrated system. Criminal history records provide considerable potential as a data source. Used in conjunction with JOLTS information, this data would provide the means to study recidivism and other issues among youth after the age of 18. In addition to checking general recidivism rates, criminal history records can also be used to check the recidivism rates of participants in various programs. Variables used to match data are name, date of birth, Social Security number, gender, and ethnicity. There are difficulties with the accuracy, timeliness, and completion of the criminal history records, an issue



Research and Evaluation: Building Infrastructure and Capacity in Arizona



14



that the criminal justice system is working to correct. This is particularly true with the disposition portion of the criminal history record.



Arizona Social Indicators

The Arizona Social Indicator profile was developed by the Arizona Department of Health Services in 2001. Social indicators are measures of behavior or experiences that characterize a risk or protective factor. Forty indicators were found to have a direct connection with substance use and behavior including arrests, violence and teen pregnancy (for a complete list see Appendix A). Social indicators potentially could be used to predict those at risk for substance abuse in specific areas in the state. In conjunction with this, social indicators can be used to provide evidence for prevention programs or areas that are in need of intervention and can indicate the type of intervention that is needed. The Arizona Social Indicator data has two primary limitations. One limitation of the Social Indicator data is that it has not been updated since 2001. With the changing demographics in the state, more current data is needed. In addition to an updated system, Social Indicator data would be valuable at the city, zip code, school and other meaningful levels of geography in order to improve evaluation capacity within the state.



United States Census

Census population data provide a wealth of demographic information, in many cases at the block level. The Arizona Department of Economic Security uses U.S. Census data in updating population estimates for each year. While very useful, these population estimates do not usually include individual age break outs by gender and ethnicity. The more detailed estimates would be extremely beneficial for research, evaluation and policy analysis, given the state's rapid population growth. This data is not routinely maintained by the Arizona Department of Economic Security. The benefit for the routine maintenance of this data for accurately assessing specific programs and related targeted populations would be significant. This is particularly relevant for the juvenile justice system, as population estimates between adult and juvenile can be quite different. It is important to note that 40 percent of the Arizona Social Indicator data is collected from census data. With any evaluation of a program, whether juvenile justice related or otherwise, reliable United States Census data will often be a necessity. The United States Census data provides critical information regarding the demographic makeup of a particular area or community. The primary limitation of both census is that it was conducted in 2000, which, given the state's population growth, may not be reflective of the current state demographics. The second limitation of this data is the difficulty in accessing and analyzing the data made available through the United States Census.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



15



The data provided by the United States Census is extremely rich, however, it is quite complex and may prevent users from benefiting from the information.



Youth Risk Behavior Surveillance System (YRBSS)

The U. S. Centers for Disease Control and Prevention developed the Youth Risk Behavior Surveillance System (YRBSS) in 1990 to monitor, on a national level, priority risk behaviors that contribute to the leading causes of death, disability, and social problems among youth and adults. These behaviors include tobacco use, unhealthy dietary behaviors, inadequate physical activity, alcohol and other drug use, risky sexual behaviors, and behaviors that contribute to violence and unintentional injuries. The Youth Risk Behavior Survey (YRBS) is administered to students in grades 9 through 12 across the country on a biennial basis. In 2003, the Arizona Department of Education began administering the YRBS to gather state-level data to track progress in meeting the State's performance measures within Arizona's No Child Left Behind consolidated application. Over 3,000 students from 87 schools participated in the 2005 Arizona YRBS. In 2003 over 15,000 students from 32 states participated in the national YRBSS survey. To access Arizona YRBS results, go to: http://www.ade.az.gov/schooleffectiveness/health/tools/YRBSSurvey.asp.



Department of Juvenile Corrections (ADJC)

The Arizona Department of Juvenile Corrections (ADJC) is responsible for juveniles adjudicated delinquent and committed to its jurisdiction by the county juvenile courts. ADJC provides valuable data on youth who are committed to its care with much of the data available on their website. Breakdowns are given for the types of crimes that juveniles are being committed for, the demographics of juveniles being committed, the number of youth in custody and how long they remain in confinement. In addition, the data summarizing incarcerated youth reentering Arizona communities can provide an element of associated risk for a given community as well as the resources necessary to address this population. Data on the offenses that high risk youth are committing, violations of parole and recidivism rates are also available. Recently the Department's Office of Research and Development, with funds from the National Institute of Corrections, received training on the application of the Correctional Program Assessment Inventory (CPAI). The CPAI is a tool that allows an agency to assess its programs and compare them with more than 300 other CPAI assessed programs. The CPAI covers six areas: 1) Program Implementation and the Qualifications of the Program Director; 2) Offender Pre-service Assessment; 3) Characteristics of the Program; 4) Characteristics and Practices of the Staff; 5) Evaluation and Quality Control; and 6) Miscellaneous items such as ethical guidelines and program stability. Each area is rated from "very satisfactory" to "needs improvement."



Research and Evaluation: Building Infrastructure and Capacity in Arizona



16



ADJC also publishes an annual outcome report providing detailed information on the success rates of parolees. This information coupled with needs information on new commitments is used to plan and program for treatment. The annual recidivism reports provide a gauge for measuring yearly progress or for comparison with the recidivism rates of other states. For more information go to: http://www.juvenile.state.az.us/.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



17



Criminal Justice Research Advisory Group (CJRAG)

From the beginning stages of the state evaluation project, it was evident that the juvenile justice system in Arizona lacked the infrastructure needed for accurate and detailed evaluations of systems and programs. This early finding provided the impetus for the Statistical Analysis Center to bring together research practitioners from various state and local criminal justice agencies to evaluate the needs of the juvenile justice system in relation to the data infrastructure needed for analyzing programs. This project sought to tap into the wealth of experience that existed among juvenile justice research practitioners around Arizona. Further, to capture diverse perspectives on juvenile justice research issues, professionals from different segments of the research community were asked to participate in a forum. This focus group became known as the Criminal Justice Research Advisory Group (CJRAG) and throughout this project they met several times to discuss issues relative to building the capacity to conduct research and evaluation on Arizona juvenile justice programs. This included representatives from academia, the Department of Juvenile Corrections, the Administrative Office of the Courts, the Governor's Office of Children, Youth, and Families, the Maricopa County Juvenile Probation Department, a private research consulting firm, the Department of Health Services Vital Statistics, and the Department of Corrections. Each of these individuals contributed their experiences in conducting evaluations on various programs in Arizona. The group discussed strategies relating to the improvement of conducting research on juvenile justice issues in Arizona. It became evident that, although there existed common obstacles among researchers, there were some challenges that had been overcome by one or more participants. In addition, there were some issues that also impacted the criminal justice system. This sharing of solutions proved to be an unexpected benefit from the periodic meetings, and benefits such as these provided the inspiration to continue to meet on a regular basis. This idea of maintaining the regular assembly of research practitioners evolved into a discussion on establishing a professional network of researchers. Bringing current projects, contemporary issues and relevant presentations to this audience was viewed as a worthwhile effort, and it was determined that assembling such an association was a good fit for the Arizona Criminal Justice Commission mission. CJRAG Infrastructure Review The main purpose of the Criminal Justice Research Advisory Group (CJRAG) initially was to investigate some of the challenges to conducting evaluations of juvenile justice programs in Arizona. As mentioned previously, during this review CJRAG found that many of the issues facing the juvenile justice system overlapped with issues facing the criminal justice system as a whole. For that reason, some of the group's focus was

Research and Evaluation: Building Infrastructure and Capacity in Arizona



18



criminal justice, as well as juvenile justice, issues. After several meetings, the team came up with a number of areas within the criminal justice system infrastructure that could be improved.

�



Availability/Collection of Needed Information



Much of the data that would assist evaluators in Arizona in their evaluation of programs is not available either because the data is not collected, the data is not distributed or made available, or because the data is collected in incompatible formats. This issue has been an acknowledged stumbling block for evaluators and researchers, and it is not easily solved. Many of the agencies that would need to collect the data or modify information systems to permit data retrieval lack the resources necessary to make these changes. The CJRAG team identified six specific areas where more information is needed: 1. Health data needs to be available at lower levels for analysis; 2. Reported crime and arrest data from law enforcement needs to be available below the agency level of reporting (i.e. by city, neighborhood, block, etc.). This would require additional data reporting and collection by agencies for lower levels of analysis; 3. Prosecution data is limited or not available; 4. Relevant data such as school or dependency needs to be collected outside the criminal justice system; 5. Juvenile gun arrest data is not available; 6. Juvenile gang arrest data is not available. It is difficult to evaluate the specific needs of a community or the effectiveness of specific programs when pertinent information for that specific community is not available.

�



Juvenile Crime Data



Related to the desire to have more information is the need to make data more available for analysis and comparison, specifically data on juvenile crime. Several factors limit the availability of juvenile data, including the need to protect the identity of juveniles and the fact that juvenile data is collected separately and in a different manner than adult data. Three specific issues were identified by the CJRAG: 1. There is a need to develop recidivism data for youth after they turn 18. This may require collecting more data, converting the data into a compatible format, and allowing identifying data to be used for analysis. There could be a potential for developing a statewide recidivism study. 2. JOLTS data should be available in sub-county level formats to increase its value for evaluation and planning.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



19



3. Additional benefits could be derived by comparing JOLTS data to UCR and NIBRS data. At the present time this is not possible. While there is a desire for more use and accessibility of data through JOLTS, it should be noted that JOLTS is a statewide juvenile probation and dependency management system that is used for a number of purposes statewide. Current functionality of the system includes: probation caseload management; detention management; billing and financial tracking; court calendaring; petition generation; victim notification; dependency, severance, and adoption tracking; treatment services tracking; diversion program tracking; juvenile tracking information into adult court; limited photo imaging; statewide e-mail; and a statewide youth index (http://www.supreme.state.az.us/jjsd/jolts/jolts.htm). It is used throughout the juvenile justice system, as well as the juvenile court's child welfare function. The CJRAG would like to see this model system be further utilized for evaluation and research purposes.

�



Attitudes Regarding Data Sharing and Evaluation



Some of the data that is currently collected is not being made available to evaluators. Some individual agencies and entities are either not willing or not able to provide information to evaluators or researchers. Various reasons exist for this issue including ownership issues, lack of certainty as to the accuracy or use of the data, and inability to access data in a format that can be shared with others. As an example, particular to juvenile data is the need to protect the identities of the juveniles. Data must often be scrubbed of names and identifying information while still attempting to maintain useable data. There is also an issue of uncertainty in some areas as to what the results of an evaluation will be, leading to a reluctance to provide data. It was also noted that there is little legislative support for research and evaluation. Currently, many policy makers view research as unimportant and unnecessary. Increasing the awareness of the value of research and evaluation at both the policy making and agency level may further develop the criminal and juvenile justice data infrastructure and increase the state's ability to evaluate programs and systems more efficiently.

�



Population



The CJRAG identified issues within the population of Arizona that make effective evaluation of programs difficult. Three issues were specifically identified: 1. In many communities in Arizona there are large populations of residents who are not represented in various sources of secondary data, such as individuals who are homeless and people who are in the country without legal citizenship. These groups are difficult to access in primary data collection efforts.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



20



2. High mobility within the lower-income population of Arizona, especially in young families with children, hinders the collection of data about that population group. 3. The fast population growth within Arizona causes data sources to quickly become outdated. There is a greater need for ongoing maintenance.

�



Limited GIS Data



One of the most impressive advances in technology has been the development of Geographical Information Systems (GIS). According to the U.S. Geological Survey, a GIS is a computer system capable of capturing, storing, analyzing, and displaying geographically referenced information; that is, data identified according to location (http://mac.usgs.gov/isb/pubs/gis_poster/#what). New applications for this tool are discovered constantly in nearly every industry of society, and GIS has added measurable value to the way a wide variety of data is analyzed. In evaluating both the effectiveness of programs and the areas that programs need to focus on, GIS can be invaluable. However, in order for this technology to be of value, various forms of data must be formatted in such a way that it is useable for the system. There is often a disconnect between the data available and the usability of the data in GIS. By transforming data into GIS layers, unrelated data, such as crime data and census data, can be analyzed. The CJRAG found a need for continued improvement of the state GIS infrastructure. Information that could be of immediate use was identified as: 1. Street layers for individual counties; 2. School attendance boundaries; 3. Expansion of efforts for disseminating data at lower levels of geography such as Community Health Analysis Areas developed by the Department of Health Services (DHS); 4. Arrest data; and 5. Service delivery program boundaries. During this study, some advances were made in developing this data. The Arizona Department of Health Services, Bureau of Public Health Statistics made significant progress in developing data sets that could be distributed using the newly created Community Health Analysis Areas that will be discussed later in this report.

�



Limited Knowledge in Conducting Evaluations



The CJRAG team found that most agencies and programs were willing to evaluate programs that have been put in place. However, there is a lack of understanding as to how accomplish this. Agencies and programs often do not possess the necessary



Research and Evaluation: Building Infrastructure and Capacity in Arizona



21



resources and/or the appropriate expertise to conduct quality evaluations. The team made the following recommendations: 1. Identify and deliver the necessary training in how to conduct program evaluations; 2. Provide technical assistance regarding how to access and use data for evaluation purposes; 3. Research and evaluation questions need to be supported by research designs that permit answering questions in a manner that provides useable data. Training and resources are needed to provide the ability to implement controlled studies; and 4. Grantees and other programs need the resources and support to implement best practice models for conducting program evaluation.

�



Partnerships



Increased partnerships among consultants, universities, the Statistical Analysis Center, and other agencies and organizations that are competent in program evaluation are needed. Given the limited resources currently available for research and evaluation, every effort should be made to take advantage of existing resources, such as data from JOLTS, the AYS, Department of Juvenile Corrections, and criminal history records, as well as other criminal and juvenile justice agencies that collect and maintain records beneficial for evaluation and analysis.



Future Plans

The Criminal Justice Research Advisory Group meetings illustrated the value of having those knowledgeable in the field of criminal justice research and evaluation meet on a regular basis for the purpose of improving the infrastructure of the criminal and juvenile justice systems in Arizona. It was determined that a more permanent venue for these meetings would provide an avenue for future improvements of the data infrastructure and would facilitate data collection and data sharing among researchers and evaluators within the criminal justice system. It would also provide the benefit of facilitating networking opportunities, allowing researchers and evaluators to have more opportunities to find out what data is available to prevent replication of projects using scarce resources. The Arizona Statistical Analysis Center is housed within the Arizona Criminal Justice Commission. The purpose of this Commission is to enhance the coordination, cohesiveness, productivity and effectiveness of the criminal justice system in Arizona. Several committees and subcommittees operate within the Arizona Criminal Justice Commission to further this goal. One subcommittee, the Technical Team of the Executive Steering Committee, meets on a regular basis to facilitate the improvement of the data infrastructure. However, this team deals with the technical aspects of



Research and Evaluation: Building Infrastructure and Capacity in Arizona



22



information technology rather than research and evaluation or policy implications of the data. It was determined that a permanent subcommittee, housed within the Arizona Criminal Justice Commission, and modeled after the Technical Team, would provide a continuing avenue for the Criminal Justice Research Advisory Group to operate. The Criminal Justice Research Advisory Group can similarly provide both the juvenile justice system and the criminal justice system feedback regarding issues relative to research and evaluation in Arizona. Composed of both Arizona Criminal Justice Commission staff and criminal justice stakeholders, this group can identify problems, make recommendations to policymakers, and facilitate information sharing.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



23



Statewide Efforts

During the course of this project, several projects were identified by the Criminal Justice Research Advisory Group as working toward the improvement of research, evaluation and data dissemination in Arizona. The list is not intended to be all inclusive but rather provides a glimpse of current efforts underway in Arizona that could be built upon. Arizona Evaluation Network (AZENet) AZENet is an organization consisting of evaluation professionals from a diverse set of disciplines in the state of Arizona. The network, designed to provide opportunities for evaluation professionals, researchers, and consultants to exchange research strategies and practical information has been an effective networking tool for more than a decade. As information gathering is typically a first step for researchers, this premise held true for those involved in initiating the Arizona Evaluation Network. From this groundwork, a forum was developed for researchers to share effective ideas and strategies for research and evaluation. Currently, through this medium, more than 100 members rely on the expertise of colleagues from around the state. In addition to its professional membership, the organization has an elected board of directors consisting of leading practitioners from the evaluation and research community and has established bylaws for effective regulation and management. Members of this association have made significant contributions in their respective disciplines and include university professors, private consultants, and public sector representatives. The organization holds semi-annual conferences, produces a periodic newsletter, maintains a membership networking directory, and conducts meetings where members can interact and benefit from shared or varied experiences. The organization is divided into clusters for more frequent and efficient networking opportunities. Regional clusters hold meetings several times a year to network which also allows a more community specific focus. Representatives from several justice agencies in Arizona belong to the network including the Arizona Department of Juvenile Corrections, Arizona Administrative Office of the Courts, Maricopa County Juvenile Probation, as well as University academicians in the justice field, which has provided an opportunity to share common experiences from the justice field. ACJC's SAC has benefited from this association, both as an agency as well as individual research personnel engaging in coordinated endeavors with other member groups. The common ground among professionals in the same field is a framework that can be built upon to address industry-specific issues, procedures that warrant change, and training opportunities. As the benefits of this organization are held in high regard by its members, this association would serve as a good model for a similar group of researchers to

Research and Evaluation: Building Infrastructure and Capacity in Arizona



24



contribute ideas for effective evaluation strategies. Many of the issues prominent within the criminal and juvenile justice communities also impact other domains such as education, health, and community planning. From the perspective of data sharing and evaluation, such a group would be valuable to building capacity for continual evaluation in the justice community. In addition, such a gathering of research professionals would be a ready forum for introducing data reporting updates, technological advancements, and resource referrals. State Epidemiological Workgroup In February 2004, Arizona Governor Janet Napolitano directed the Arizona Drug and Gang Policy Council (ADGPC) to create a statewide, data-driven strategic plan on substance abuse prevention needs and services. This project involved state agency leaders, community members and the Governor's cabinet representatives. Using the Strategic Prevention Framework State Incentive Grant, several working groups were created to address separate objectives of the initiative. One of the major stakeholders in this project is the Resource Management System Data Subcommittee, which serves as the State Epidemiological Workgroup (SEW). The SEW's purpose was to: o bring systematic, analytical thinking to the causes and consequences of the use of substances in order to effectively and efficiently utilize preventions resources o promote data-driven decision making at all stages in the Strategic Prevention Framework o promote cross systems planning, implementation and monitoring efforts o provide technical support to the SPF Advisory Council The Arizona SAC participates in the Statewide Epidemiological Workgroup which operates under a Strategic Prevention Framework (SPF) State Incentive Grant (SIG). An objective of the workgroup is to create a plan to study a variety of social indicators in a specific but practical manner. According to the SPF SIG Advisory Council, "One of the major goals of this initiative was to enhance the infrastructure to increase Arizona's capacity to implement, sustain, and improve effective substance abuse prevention services (Statement of Need section, 2005). In order to accomplish this goal, resources and funding from many separate entities was necessary. Typically, current statewide substance abuse prevention funding in Arizona comes through a variety of sources including the Governor's Office for Children, Youth, and Families, Arizona Department of Education, and the Arizona Department of Health Services. In an effort to better coordinate funding and services, the ADGPC compiled data from recent large scale studies on substance abuse including the Arizona Youth Survey, Social Indicators Study, Youth Risk Behavior Study, Monitoring the Future Study, and National Survey on Drug Use and Health. Together, this information portrayed what issues needed to be addressed, what programs are available, where prevention resources are being spent, the effects of those programs, and where to direct future efforts.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



25



Segments of this model, while specific to substance abuse for this program, provide a solid foundation for other similar types of programs. These outcomes include providing a method of uniform data collection and analysis across the state that matches program services and needs, developing a set of guiding principles and a framework for datadriven policy and program development across state and local agencies, targeting resources to areas with the greatest need, and matching prevention needs and services to prevalent risk and protective factors (SPF SIG Resource Manual, section A).



Community Health Analysis Areas (CHAAs)

There are several challenges of presenting data at a sub state level through geographic analysis. It is critical that the region is large enough so as not to violate confidentiality constraints but also small enough to provide a meaningful representation of the population for a region. Part of SEW's plan is to capitalize on an innovative model called Community Health Analysis Areas (CHAA) used to study cancer episodes around the state, and transfer it into one that can be used to study social indicators in the same manner. Taking advantage of the expertise behind the CHAAs, the workgroup plans to incorporate the strengths of this model into one that could possibly serve a wide variety of disciplines in the criminal and juvenile justice communities.



Arizona Primary Care Areas (http://www.azdhs.gov/hsd/profiles2005/index.htm)



The Arizona Department of Health Services- Bureau of Public Health Statistics (BPHS) recently developed a method of defining community boundaries in an effort to research the incidence of cancer in Arizona residents. Immediately understanding that state and even county aggregations were not going to provide the most effective data, the department sought to define communities at the most precise level of demographics as possible. While many states use zip codes and county boundaries for the purposes of analyzing health and behavioral-related issues, it was determined that these areas of analysis would not be suitable for Arizona for two primary reasons. The state is currently experiencing an explosive population growth that causes frequent changes to zip code boundary definitions, impacting the stability of zip codes over time. Generally, most communities are not organized by geographic boundaries consistent with the boundaries of zip codes. Also, because there are relatively few counties in Arizona, large numbers of communities are combined and represented together, which prohibits a level of community grouping specific enough to allow for critical factors to be considered.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



26



States with stable populations are better able to use zip codes for community analysis. Because Arizona is the second-fastest growing state, the continual increase in population makes using zip codes impractical, as existing ones change and new ones are added. After considering several options, the BPHS generated areas from currently existing Primary Care Areas (PCA). PCAs have been used to characterize the health needs of communities for many years; however, the PCAs did not quite provide all the coverage and detail that was required of the CHAAs (http://www.azdhs.gov/phs/azchaa/chaa_q_a.pdf, pg. 1). The challenge was to discover a geography "that both defines communities and has a sizable population denominator (DHS PowerPoint, Community Health Analysis Areas, 2004)." The CHAAs are considered to be an improvement over the PCAs because they have fixed unchanging boundaries and a population size large enough to do analysis but small enough to represent a community. CHAAs also more closely aligned to the growing rural communities and to Phoenix villages (CHAAs FAQs). A CHAA is built from 2000 U.S. Census block groups, which are relatively small geographic regions of the state. A typical CHAA contains approximately 21,500 residents, but the range varies widely from 5,000 to 190,000 persons. The use of the relatively small block groups provides the ability to aggregate data at a variety of sizes, from the block group level up to the CHAA level for analysis. This system represents geographic areas that are small enough to be representative of the community, yet large enough to generate a substantive denominator. This fixed boundary design will allow the comparison of samples taken at different times to observe any occurring trends.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



27



Developing a system to study a specific characteristic of a population gains immediate credibility when based upon a fundamentally sound model. Just as the CHAAs evolved from PCAs, the SEW hopes to apply the same logical process to transform the cancerspecific model of the CHAA to one that can analyze other social components. Another hopeful outcome is to produce a system that looks at substantive but specific populations relating to a variety of social indicators. IMAP In 2004, the Arizona Statistical Analysis Center applied for a grant from the Bureau of Justice Statistics to fund a project that would provide a variety of data online and in an interactive map format, making it both readily accessible and easily understood. This project had two primary purposes; the first was to increase the availability of information for criminal justice practitioners, and the second was to increase the GIS capabilities of the SAC. The Bureau of Justice Statistics approved the grant, and the project was named the Internet Mapping and Analysis Project (IMAP) (http://azcjc.gov/sac/imap.asp). In concept, the purpose of the project was to take information currently available from different data sources, and put it into a geographical information system (GIS). This data could then be manipulated by the user to create maps based on their particular needs. IMAP was to serve as a medium through which the delivery of inter-related data could be presented. IMAP provides the means for the presentation and analysis of diverse data, exploring relationships across multiple disciplines. Designed to assimilate

Research and Evaluation: Building Infrastructure and Capacity in Arizona



28



multiple data forms that might not otherwise be used together, IMAP makes data available to those interested in social or criminal justice issues. This feature also provides the added benefit of flexibility necessary to make this information available to communities, schools and agencies ranging in size and purpose. SAC continually strives to find new ways to put information into the hands of practitioners and policy makers in such a manner that it can be easily understood and applied. Their involvement in numerous collaborations has increased both the awareness of available data and the opportunities to share that information. SAC has taken advantage of current technology in providing criminal justice professionals access to information in an interactive format. In so doing, the GIS tool equips the user to come away with specific data relevant to their area of interest or service. These coordinated efforts involve government, academic and private research professionals who work together capturing, analyzing and sharing a wide array of data. The overarching mission of these collaborations is to improve both the identification of criminal and social concerns and the delivery of programmatic solutions. The initial version of IMAP was intended to demonstrate the possibility of creating a single source for a wide array of data. It is the SAC's intent to leverage this data to serve two purposes: to provide information that is relevant to agencies involved in similar practices, and to portray this information in a geospatial format that effectively illustrates data relationships. Examples of this would include determining where at-risk populations are concentrated and where services or treatment centers are located in relationship to one another. The IMAP wizard, available on the Arizona Statistical Analysis Center's web site at www.azcjc.gov/sac/imap.asp, allows users access to Uniform Crime Report, Arizona Youth Survey, and Census data. Information is available in two formats through this tool. In the GIS map format, users select the information that they want to view in order to create a customized map showing the information selected. In the downloadable files, the user selects the information needed, which results in an Excel file being made available for download. These files also contain bars and charts that are available for download and use. The map format allows the user to get a quick snapshot of the data, while the downloadable files allow for more in-depth analysis. Data is available at the county level.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



29



The two screenshots above show an example of the type of information that can be presented through the IMAP wizard. In the first screen, a map was created showing 30day marijuana use in 2004 for eighth grade students. The following screen shows the file that was created using the wizard that includes all drug use in Maricopa County in 2004. This tool can be used to create similar maps and files for all of the possible selections. Introducing the IMAP concept is only the first step toward improving the criminal justice industry's ability to share and analyze data. This technology is located at the ACJC

Research and Evaluation: Building Infrastructure and Capacity in Arizona



30



facility and managed by SAC and ACJC's Information Technology staff. Having ready access to the application will provide opportunities to improve functionality and add valuable data related to future collaborations. The long-range goal of the project is to maintain a robust tool that adds value to the research community across the broad range of specialization. The Criminal Justice Research Advisory Group worked closely with the Statistical Analysis Center during the testing phase of this project to pilot test the program and evaluate its effectiveness. Suggestions for improvement from the group resulted in changes that were made prior to the implementation of the program on the Statistical Analysis Center web site. Additional functionality will be added to the site upon the roll out of the data for the upcoming year. Center for Violence Prevention and Community Safety A new initiative based at Arizona State University West is the establishment of Center for Violence Prevention and Community Safety. The Center, housed in College of Human Services, is a comprehensive multidisciplinary research development center focused on gathering and providing progressive findings strategies to prevent violence and improve community safety. the the and and



The Center, which is directed by Dr. Charles Katz, professor of criminal justice and criminology, is described by the university president as "one university in many places." This essential repository is designed to concentrate research, education, and service toward a distinct focus. It is unique in design in that it draws not only from multiple academic disciplines, but brings community and private organizations and government agencies into partnership to confront violence related issues. The Center for Violence Prevention and Community Safety has considerable potential as a resource for addressing violence and working toward safer communities in Arizona. As social and behavioral scholars network to discover and apply best practices to affect community safety, this innovative approach could be replicated to address regionally specific issues nationwide. The Center provides a forum for idea exchange and implementing successful strategies, as well as a launching point for practitioners and academicians to align together for specific endeavors. The Center's synergistic approach to relevant community and behavioral matters could make it an important resource for the entire state.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



31



Re-Entry Council The issue of successful reentry of incarcerated persons, whether juvenile or adult, into the community is one with far-reaching implications. If successful, the person can move on to productive activity. However, if unsuccessful, the toll on the community and the criminal justice system is significant. A national Reentry Council was formed by the Council of State Governments to study how states can work to ensure successful reentry. The Report of the Re-Entry Policy Council made a number of recommendations. A primary recommendation was that reentry services be located in the areas that probationers, released prisoners and parolees are concentrated in order to maximize the effectiveness of limited resources. The study showed that these people tend to live in concentrated areas, but that services geared toward them are frequently not near them. Services include drug treatment, treatment for physical and mental illnesses, job training, probation and parole office, and others. The end goal of the project was to enable states to make smart release and community supervision decisions that ensure support for victims, while enabling those reentering society to succeed. In Arizona, a taskforce created by the Arizona Department of Corrections and the Governor's Office called the Reentry Council is currently working to integrate and analyze information from several contributing criminal justice agencies. This Council is modeled after the national Re-Entry Policy Council. These agencies include but are not limited to the Arizona Department of Corrections, the Arizona Department of Juvenile Corrections, and Maricopa Juvenile Probation Department. The Reentry Council is working with experts in this field to develop a model for assessing high impact areas upon the Arizona criminal justice system. Dr. Eric Cadora, head of Justice Mapping, and Dr. James Austin, a renowned criminologist, both consultants to the Council for State Governments are working with the Reentry Council in this endeavor. This project will make use of geographic information systems (GIS) in research and practice in the corrections system, including probationers, prisoners and parolees. Data mapping analysis done through the national Re-Entry Policy Council has shown that parolees, probationers, and persons entering and leaving prison are highly concentrated in certain neighborhoods in larger cities, and that these neighborhoods also have high concentrations of singe parent households, youth, and people on various public assistance programs. Research indicates that not only do poverty and poor family structure lead to crime, but that concentrations of criminals contribute to poverty and familial deterioration. On the Reentry Council are representatives from the Departments of Correction and Juvenile Corrections, Maricopa County Juvenile Probation, the Governor's Office, and ACJC. The Arizona Reentry Council is identifying these neighborhoods in Arizona so criminal justice and health and human services can be structured to serve them more efficiently and effectively. GIS is being used to evaluate the prevalence of probationer, prisoners



Research and Evaluation: Building Infrastructure and Capacity in Arizona



32



and parolees per 100,000 residents at the zip code and county levels in Arizona. It is also being used to identify areas with high concentrations of youth, residents on public assistance and minorities. This will identify the areas most in need of intervention. It is anticipated that this long-term strategy will significantly reduce expenditure related to criminal justice in these areas.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



33



Survey Interviews of JABG, Weed and Seed Sites, and Drug Free Community Coalitions

It was the collective experiences of the members of the CJRAG that there are many hurdles in the path toward good evaluation. Collaborating with the CJRAG and the Governor's Office for Children, Youth and Families (GOCYF), SAC staff surveyed staff from JABG programs, Weed and Seed Sites, and Drug Free Community Coalitions throughout Arizona, and also reviewed numerous quarterly reports from JABG and JJDP subgrantees submitted to the GOCYF from October 2002 through March 2004. The intent of the surveys and the review of the quarterly reports was to discover the experiences of program personnel in collecting data and using it to demonstrate the effectiveness of their programs. The survey was developed by SAC staff and conducted over the phone. It contained nine questions focusing on how the program demonstrated its effectiveness, issues involving data collection, and the value of the program to the particular jurisdiction the program served. At the end the respondents were asked for additional comments. There were slight variations in the survey administered to the JABG subgrantees. For each program that was surveyed, SAC staff interviewed an individual who was identified as a "program contact" in the grant's description. Similarly, the review of the quarterly reports focused on the collection of data and what additional information could be provided to demonstrate effectiveness. JABG Subgrantees In Arizona there is a great deal of diversity in the size and focus of individual JABG programs. The programs surveyed for this project received funds ranging from less than $6,000 to more than $950,000. This diversity is to be expected, given the structure of the grant with its array of program purpose areas and allowance for local areas to receive funding for very specific local issues. Although subgrantees retain local decision making authority regarding grant funds, this is limited to expending funds within preset program purpose areas. JABG provided funds for purchasing computer equipment and drug screening apparatus; for hiring judges pro tem, additional probation staff, and additional prosecutors. JABG funds were used to provide educational programming for juveniles in a rural Native-American detention center, and for programs that reduced domestic violence in a large urban inner city area. They funded juvenile accountability and skill development programs. The juveniles served by these programs ranged from hard core delinquents to first time truants, to grade school age students identified as potential gang members. Respondents from 21 of the 27 Arizona programs participated in the phone survey. The survey consisted of nine open-ended questions. For some respondents some of the nine questions were modified in process or re-sequenced to fit the context of the interview.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



34



Because of the diverse nature of the programs, some agencies reported that they were able to easily evaluate and show effectiveness, while other agencies believed their programs were ended because of an inability to demonstrate effectiveness to federal funding sources. Problems in demonstrating effectiveness cover a broader spectrum than the Weed and Seed or Drug Free Communities programs. Several had the typical problems of scant resources (time, money, or staff). But the rest had problems unique to their program: measuring the effect of installing computer equipment; measuring a community's more positive perception of probation; having to provide services to a population that was not the ideal population for the service provided; continuity problems when service providers merged with other agencies; logistical problems with testing juveniles who entered and left the program at different times. Almost all programs were able to produce the data requested for progress reports, although there was some difficulty with JABG performance measures not fitting neatly with local programs. It is important to note that rule changes pertaining to performance measures on specific purpose areas were implemented mid-stream in the granting cycle. This presented numerous challenges for subgrantees in providing information in meeting reporting requirements set by federal funding sources. All of the respondents believed their program was very valuable to their jurisdiction, and that if funding was discontinued the service and value would be lost. Some of the values provided to the communities were: holding juveniles accountable and increasing their compliance with court orders; providing something more than a punitive response to breaking the law; providing a community with its only after-school drug prevention program; decreasing calls for police service in a highly delinquent area; improving relations between a probation department and schools, improving the exchange of information among police agencies and enabling them to comply with FBI requirements for reporting crime; increasing collaboration between independent agencies. Virtually all of the respondents expressed disappointment that their program was losing funding and that it would be very unlikely that the services could be continued. In a few instances they believed they had even demonstrated the program's effectiveness (via recidivism rates). Across the board, subgrantees reported that the Governor's Office had worked well with them in setting up programs and benchmarks. Drug Free Community Coalitions Drug Free Community Coalitions are the product of the Drug Free Communities Act of 1997. This Act was intended as a catalyst for increased citizen participation in efforts to reduce substance use among youth, and it provides community anti-drug coalitions with much-needed funds to carry out their important missions. The White House Office of National Drug Control Policy directs the Drug-Free Communities Support Program in partnership with the Substance Abuse and Mental Health Services Administration. The program provides grants of up to $100,000 to



Research and Evaluation: Building Infrastructure and Capacity in Arizona



35



community coalitions that mobilize their communities to prevent youth alcohol, tobacco, illicit drug, and inhalant abuse. The grants support coalitions of youth; parents; media; law enforcement; school officials; faith-based organizations; fraternal organizations; state, local, and tribal government agencies; healthcare professionals; and other community representatives. The Drug-Free Communities Support Program enables the coalitions to strengthen their coordination and prevention efforts, encourage citizen participation in substance abuse reduction efforts, and disseminate information about effective programs. Representatives from all nineteen of the Drug Free Community Programs participated in the survey. The challenges they faced in measuring effectiveness were: lack of resources (time, staff, funds, or expertise); reporting requirements changing from one year to the next; difficulty of establishing measurements for some types of services (e.g. measuring the effect of pamphlets and posters); logistical problems in gathering longitudinal data (people moving); isolating the effect of your specific program from an array of other influences on juveniles. All the programs needed adult and/or juvenile criminal justice information. Most of them were getting it, but not without difficulty: "it was too hard to get; it could not be gotten at the local level; or the program's service area did not match the area covered by the data they were given." There was widespread reliance on the data provided by the Arizona Youth Survey for areas where it was available (about half of them). In the spring of 2005, SAC staff provided several coalitions with tabular data showing the prevalence of ATOD and related risk factors for use among students that participated in the AYS in their area. SAC gave these requests high priority and were able to achieve all data requests. Many programs stated that they needed AYS data to complete required progress reports. About half of the programs had conducted an evaluation of their program and it was usually done by contract with an outside agency. However, other coalitions noted a lack of resources and staffing as well as other difficulties that prevented them from evaluating programs to the degree they felt was needed. Common difficulties included a lack of resources and staffing, a lack of the data needed, especially juvenile data, and a lack of long-term access to the youth they were working with. Given the preventive aspect of these programs, coalitions noted the need to track youth over time in order to determine the long-term effectiveness of programs, something they lacked the resources and access to accomplish. Other difficulties noted included a lack of knowledge in how to conduct evaluations. Other expressed problems and needs were: mandated collection of data that was not relevant or reliable; transition from the ONDCP model to the SAMSHA model; lack of money earmarked for evaluation; increasing collaboration with other agencies, using



Research and Evaluation: Building Infrastructure and Capacity in Arizona



36



the Statistical Analysis Center at the Arizona Criminal Justice Commission as a resource; long term tracking of juveniles (especially those on prevention programs). Weed and Seed Weed and Seed is a grant program funded by the U.S. Department of Justice Office of Justice for Community Based Initiatives. It is an innovative and comprehensive approach to law enforcement, crime prevention and community revitalization. Operation Weed and Seed is foremost a strategy--rather than a grant program--which aims to prevent, control, and reduce violent crime, drug abuse, and gang activity in targeted high-crime neighborhoods across the country. Weed and Seed sites range in size from several neighborhood blocks to 15 square miles. The strategy involves a two-pronged approach: law enforcement agencies and prosecutors cooperate in "weeding out" criminals who participate in violent crime and drug abuse, attempting to prevent their return to the targeted area; and "seeding" brings human services to the area, encompassing prevention, intervention, treatment, and neighborhood revitalization. A community-orientated policing component bridges weeding and seeding strategies. Officers obtain helpful information from area residents for weeding efforts while they aid residents in obtaining information about community revitalization and seeding resources. All of the nine Weed and Seed programs in Arizona participated in the survey. Most of the programs had conducted some form of evaluation. The challenges they faced in measuring effectiveness included: cost and/or lack of funding, difficulty in finding needed information, getting approval for using data from other agencies, and accessing or establishing baseline data. One agency had staff turnover which led to a change in focus, caused partially by lack of baseline data. Programs that contracted with smaller non-profit agencies reported that the skill level of those collecting or reporting the data varied. Another reported that the program's focus led to increased reporting of the behavior to police. There was a consistent belief that juvenile and adult criminal justice information was valuable. About half of the programs had access to it, but still found it problematic because they could not get information for their service area, or it was difficult to get, or juvenile data could not be separated from adult data. In addition to not having criminal and juvenile justice data, the next most desired type of data needed was drug related data. Other expressed needs or problems were: inability to report anecdotal information in the reporting format; the phenomenon of the "problem behavior" leaving the targeted area and moving to an adjacent area; and having state-wide meetings of coalitions to share data-related problems. A typical example of anecdotal information that showed success but was not able to be reported expressed was "one area had trick-or-treaters



Research and Evaluation: Building Infrastructure and Capacity in Arizona



37



for the first time in over a decade, where do you report that." Finally, demonstrating success to those who were not part of the program concerned many respondents. They were frustrated with the lack of resources and ability to do it, and also with a lack of support and understanding of why it was needed.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



38



Findings

Data strengths 1. Arizona has extremely strong data on risk and protective factors and the prevalence of substance abuse and gang behavior of juveniles. The Arizona Youth Survey provides data at the state and local levels in Arizona. 2. The Internet Mapping Analysis Project (IMAP) is an innovative web site program developed by ACJC that takes information currently available from different data sources and puts it into a geographical information system (GIS). 3. JOLTS is a nationally recognized comprehensive statewide database, that has been replicated in several other jurisdictions in the country. Data Limitations 4. Arizona has very limited data regarding the prevalence of adult substance abuse. 5. Data that can be used to track recidivism on youth beyond their 18th birthday needs to be more accessible. 6. Juvenile gun arrest data is not available. 7. Juvenile gang arrest data is not available. 8. Reported crime and arrest data from law enforcement needs to be available below the agency level of reporting (i.e. by city, neighborhood, block, etc.). 9. Reliable data regarding victim-related crime is not available (domestic violence, etc.). 10. Prosecution data is limited or not available. 11. Data collection, reporting, and dissemination within the Arizona Criminal History Records Program have considerable room for improvement in the completeness, accuracy, timeliness and accessibility of data for the purpose of research and evaluation. 12. The fast population growth within Arizona causes data sources to quickly become outdated requiring increased resources dedicated to ongoing maintenance.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



39



Inter agency support and collaboration 13. Feedback from JABG subgrantees expressed extremely positive feedback regarding the support and communication they receive from the Governor's Office. 14. There is not currently an infrastructure in place for researchers within the criminal and juvenile justice system to network and exchange information. 15. Statewide JOLTS data is not readily available in sub-county level formats that would increase the value of JOLTS as a tool for evaluation and planning. 16. Subgrantees surveys indicate there is a widespread lack of access to data necessary for reporting purposes and evaluating their programs. 17. There are often challenges associated with disseminating data for the purpose of evaluating programs due to confidentiality issues and sensitivity of host agencies. Infrastructure 18. Although Arizona has very strong data systems, efforts to use these data sources effectively are only just beginning. 19. Programs often lack the resources necessary to conduct evaluations. 20. Programs often lack the knowledge of how to conduct an evaluation. 21. There is insufficient awareness of the need to conduct evaluations and demonstrate the effectiveness of programs within Arizona. 22. Subgrantees report there is little time and resources for the purpose of research and evaluation. 23. The AZENet is a group of researchers in Arizona that provides opportunities to share resources and network about issues related to evaluation. ASU's Center for Violence Prevention is an innovative "one-stop" resource for research and may be a valued partner for research and building capacity for evaluation. 24. Surveys found a general lack of awareness of available resources in the state such as: social indicator data, crime data, and the Arizona Statistical Analysis Center. 25. Arizona does not have a state plan for building the capacity for conducting evaluations.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



40



Recommendations

Developing Awareness for Evaluation 1. 2. 3. There needs to be an increase in the awareness of the importance of evaluating programs and projects in order to demonstrate their effectiveness in Arizona. Increased support and resources should be dedicated to research and evaluation in Arizona. Arizona should develop a state plan for building the capacity to conduct research and evaluation. The plan could be developed and maintained by the Juvenile Justice Commission and the Arizona Criminal Justice Commission. A statewide meeting and strategic planning session of Drug Free Communities should be held to increase networking efforts, information sharing, and knowledge transfer. Increased resources should be directed to the provision of training and technical assistance for local programs on how to access and use data for evaluation purposes. Research and evaluation questions need to be supported by research designs that permit answering questions in a manner that provides useable data. Additional training and resources are needed to provide the ability to implement controlled studies in Arizona.



4.



5.



6.



Increasing Data Sharing and Accessibility 7. 8. The data infrastructure in Arizona needs to be enhanced to allow agencies the ability to share information across systems. Increased priority should be given for sharing information between criminal and juvenile justice systems for the purpose of research, evaluation and policy analysis. Arizona should develop strategies for increased access to vital juvenile and adult arrest data for the purposes of research, evaluation and policy analysis.



9.



Expanding the Data Infrastructure 10. Given the impact of juvenile gun and gang related crime, data regarding these issues should be collected within juvenile information system (JOLTS).



Research and Evaluation: Building Infrastructure and Capacity in Arizona



41



11. Increased efforts and resources should be dedicated toward developing data extracts allowing reported crime and arrest data from law enforcement to be available for research and analysis below the agency level of reporting (i.e. by city, neighborhood, block, etc.). 12. The availability of prosecution data for Arizona is quite limited. Arizona should develop common definitions for prosecution and create a data extract that could be used for research, evaluation and policy analysis. 13. There is a need to develop recidivism data for youth after they turn 18. This may require collecting more data, converting the data into a compatible format, and allowing identifying data to be used for analysis. There could be a potential for developing a statewide recidivism study. 14. The fast population growth within Arizona often results in data sources to becoming quickly outdated. There is a greater need for ongoing maintenance. 15. GIS information that would be of immediate use to evaluators was identified by CJRAG as: street layers for individual counties; school attendance boundaries; expansion of Community Health Analysis Areas developed by the Department of Health Services (DHS); arrest data; and service delivery program boundaries. 16. Health data such as emergency room admittances, gunshot wounds information, drug overdose data and mortality data should be available at lower levels for analysis.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



42



Conclusion

Although it is clear that Arizona has much work toward building the capacity for conducting research and evaluation within the criminal and juvenile justice systems throughout the state, a strong foundation from which to build upon has already been developed. Two of the tools that are most utilized within the juvenile justice system are the Juvenile On-Line Tracking System and the Arizona Youth Survey. By collecting data on youth already in the juvenile justice system and on youth as a whole, these two sources have the capability to provide a wealth of data. However, both of these tools need to be made available to a wider range of programs and at levels that can be used by programs. Breakdowns at the state and county level are not sufficient for many of the programs to use to evaluate their programs. It is important to note that although Arizona has room for improvement in the area of evaluation, many projects are currently in the works to improve the integration of systems and provide programs the ability to share information. A couple of these initiatives include the State Prevention Framework-State Incentive Grant (SPF-SIG) grant and the Reentry Council Task Force. The SPF-SIG grant in Arizona is working to fund the integration of systems addressing substance abuse and related issues in Arizona. This is a five year project that will look at building the overall infrastructure and related data systems for collecting and sharing information. The Reentry Council, a taskforce created by the Arizona Department of Corrections and the Governor's Office is currently working to integrate and analyze information from several contributing criminal justice agencies. The concepts of both these projects are to use information systems for assisting policy makers with decisions for directing limited funding dollars. These projects are also striving toward improving data collection strategies and the availability of these data systems. Other projects within the justice system include the NCHIP and CJRIP grant programs that fund integration programs statewide. These projects, and others like them, can build upon previous successes in the state such as the statewide implementation of the JOLTS system. As part of the partnership with the Governor's Office for Children, Youth and Families, Juvenile Accountability Block Grant subgrantees were engaged through phone surveys to ascertain challenges in evaluation and reporting on progress. Subgrantees that were surveyed felt very strongly that JOLTS provided them with the majority of the data they used that was outside program-collected data. While respondents whose grants were ending indicated frustration with the lack of understanding on the federal level as to the value of the funding, grantees overwhelmingly indicated a large degree of satisfaction with the Governor's Office and their helpfulness with the grant process. In order to fulfill the goal of increasing cooperation and evaluation capacity throughout the state, the Arizona Statistical Analysis Center worked with Weed and Seed and Drug Free Community grantees to provide them with specialized data sets from the Arizona Youth Survey specific to their area of concentration. Those surveyed felt this data were

Research and Evaluation: Building Infrastructure and Capacity in Arizona



43



invaluable in providing the information they needed to report progress to funding agencies. Respondents also expressed that prior to Arizona Youth Survey data being provided to them, many were not able to complete reporting requirements as established by their individual funding sources. The Arizona Statistical Analysis Center plans to continue working with these groups to improve evaluation capacity within their programs. A Criminal Justice Research Advisory Group was formed through this process. This group formulated recommendations on improving the evaluation infrastructure throughout the state, as well as helped review a number of projects. Members of this group evaluated the Arizona Criminal Justice Commission's Internet Mapping Analysis Project (IMAP) prior to its release, as well as the Arizona Youth Survey and conducted an infrastructure review of needs throughout the state. It was clear that the benefit of this group went beyond the purposes of this project. It is recommended that members of this group continue to meet and become a working subcommittee of the Arizona Criminal Justice Commission. Along with increasing the value of JOLTS and AYS data, juvenile justice evaluators throughout the state cited the need for better access to Geographic Information System (GIS) data. This source of information, while extremely valuable for its ability to put information into formats useable for a variety of needs, is in its infancy. A beginning attempt to make information available to programs using a GIS format, the Internet Mapping Analysis Project was worked on by the Criminal Justice Research Advisory Group during this project. IMAP took data from UCR, AYS, and the US Census to create maps that the user could generate based on specific needs at the state and county level. This first stage showed the usefulness of putting large amounts of data into a common format so that they can be viewed together. A GIS infrastructure needs to be developed throughout the state at levels that maintain confidentiality where needed, but provide information at lower levels of analysis. This data infrastructure, if maintained and made accessible, would provide a wealth of information for researchers and evaluators. Evaluations are most effective if programs are designed with evaluation components built in. This requires that funding agencies and programs understand the need for and benefits of evaluating programs. Requiring that funds be set aside for evaluation for programs statewide would allow programs to be evaluated, and for best practices to be highlighted for replication where appropriate. While the majority of programs surveyed for this project understood the importance of evaluation, surveys indicated a need to heighten awareness of the need for evaluation, as well as the resources available for evaluating programs. Many program administrators recognized the importance of program evaluation, but either lacked the expertise or finances necessary to conduct evaluations.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



44



While Arizona has a wealth of data available, data is currently collected in ways that are not necessarily comparable with other forms of data. Some data is available at the individual level, while others are collected at zip code, school boundary, census tract, city, county or state levels. Programs desiring to use multiple data sources to evaluate effectiveness often are unable to get data relating to the specific area they serve, limiting the usefulness of the data. A statewide plan to standardize data sources and provide for the evaluation of all programs would allow programs to conduct meaningful evaluations. State agencies need to come together to formulate a statewide plan for evaluation that will give both state and local entities the guidance needed to go forward. Along with the need for a state plan is the need for legislative support for evaluation. More importantly, states should develop a strategy/plan in which resources are dedicated toward improving data sharing, data collection, and evaluation. This project served as a catalyst for improving relationships between agencies and enhancing the understanding of the need for evaluation throughout the state. Existing means from which to build a solid infrastructure include the many data systems and entities mentioned previously in this report, as well as the many agencies and programs that work peripherally with the justice system. Both the Arizona Criminal Justice Commission and the Juvenile Justice Commission serve as coordinating and facilitating agencies within Arizona and might be an opportunity for these entities to work more closely together. Certainly key stakeholders, such as the Criminal Justice Research Advisory Group, should be brought together in the development of this strategy. As evaluation became a national priority, the Justice Research and Statistics Association (JRSA) commissioned a project that evaluated how states approached the evaluation of programs. This project highlighted two states that have created a statewide plan that requires as a matter of policy that state monies be set aside for evaluating programs. In some cases, outside evaluators were required. In all cases, the focus was on demonstrating the effectiveness of the programs. Evaluations were then used to set spending priorities, highlight model programs, and drive policy. Regardless of the specific strategy employed, the emphasis on evaluating programs statewide allowed the states to ensure that limited funds were spent effectively. Arizona has the data infrastructure from which to build upon to become a leader in evaluating programs. If made a priority throughout the state, effective programs in Arizona can be recognized for their success. Existing data must become more accessible and interconnected in order to allow programs to effectively ascertain their successfulness. Decision makers are encouraged to utilize the Research and Evaluation: Building Infrastructure and Capacity in Arizona report in developing strategies for improving the criminal and juvenile justice data infrastructure and for assuring that adequate resources are dedicated toward research and evaluation in Arizona. Only through such an approach can policy makers be provided information to develop policies that have the greatest impact in the areas that are most in need of programs.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



45



Bibliography

Arizona Department of Health Services. Arizona Primary Care Area Program. http://www.azdhs.gov/hsd/profiles2005/index.htm. Internet.



Arizona Department of Juvenile Corrections. http://www.juvenile.state.az.us. Internet.

Arizona Department of Juvenile Corrections. 7th Annual Outcome Evaluation, August 2003. www.juvenile.state.az.us/Offices/Research/Publications/PdfFormat/ 7th%20Annual%20Outcome%20Eval%20Report.pdf. Internet. Arizona Department of Juvenile Corrections. 6th Annual Outcome Evaluation, August 2002. www.juvenile.state.az.us/Offices/Research/Publications/PdfFormat/ 6th%20Annual%20Outcome%20Eval%20Report.pdf. Internet. Arizona Office of the Courts. Juveniles Processed in the Arizona Court System FY04. Arizona Statistical Analysis Center. Arizona Criminal Justice Commission Internet Mapping & Analysis Project. www.azcjc.gov/sac/imap.asp. Internet. Arizona Statistical Analysis Center. Arizona Youth Survey 2002. Arizona Criminal Justice Commission. Arizona Statistical Analysis Center. Arizona Youth Survey 2004. Arizona Criminal Justice Commission. Arizona Statistical Analysis Center. Crime Trends: A System Review. Arizona Criminal Justice Commission. 2002. Arizona Statistical Analysis Center. The Internet Mapping and Analysis Project: Final Report. Arizona Criminal Justice Commission. 2005. Arizona Supreme Court. Juvenile On-Line Tracking System (JOLTS). http://www.supreme.state.az.us/jjsd/jolts/Default.htm. Internet. Barnett, L., Parent, D. Juvenile Accountability Incentive Block Grant Program: National Evaluation. ABT Associates. 2003. Bureau of Justice Statistics. National Incident-Based Reporting System (NIBRS) Implementation Program. http://www.ojp.usdoj.gov/bjs/nibrs.htm. Internet. Centers for Disease Control and Prevention. YRBSS: Youth Risk Behavior Surveillance System. http://www.cdc.gov/HealthyYouth/yrbs/index.htm. Internet.



Research and Evaluation: Building Infrastructure and Capacity in Arizona



46



Division of Criminal and Juvenile Justice Planning and Statistical Analysis Center. Juvenile Justice System Evaluation Capacity-Building in Iowa. Iowa Department of Human Rights. 2002. Federal Bureau of Investigation. Uniform Crime Reports. http://www.fbi.gov/ucr/ucr.htm. Internet. Gottfredson, D., Povitsky, W. An Evaluation Partnership Project to Enhance the State of University of Maryland: Department of Criminology and Criminal Justice. 2005.



Maryland's Capacity to Evaluate J