Special study, baseline study of Arizona's English language learner programs and data, fiscal year 2007

              Special Study
Baseline Study of Arizona’s
English Language Learner
Programs and Data
Fiscal Year 2007
Division of School Audits
Debra K. Davenport
Auditor General
APRIL • 2008
A REPORT
TO THE
ARIZONA LEGISLATURE
The is appointed by the Joint Legislative Audit Committee, a bipartisan committee composed of five senators
and five representatives. Her mission is to provide independent and impartial information and specific recommendations to
improve the operations of state and local government entities. To this end, she provides financial audits and accounting services
to the State and political subdivisions, investigates possible misuse of public monies, and conducts performance audits of
school districts, state agencies, and the programs they administer.
Representative John Nelson, Chair Senator Robert Blendu, Vice Chair
Representative Tom Boone Senator Carolyn Allen
Representative Jack Brown Senator Pamela Gorman
Representative Pete Rios Senator Richard Miranda
Representative Steve Yarbrough Senator Rebecca Rios
Representative Jim Weiers (ex-officio) Senator Tim Bee (ex-officio)
Ross Ehrick, Director
Tara Lennon, Manager and Contact Person
Briton Baxter Thomas Huber
Patricia Beckman Melissa Redding
Christopher DeSanto John Ward
Bryant Duffek Jennifer Wild
Copies of the Auditor General’s reports are free.
You may request them by contacting us at:
Office of the Auditor General
2910 N. 44th Street, Suite 410 • Phoenix, AZ 85018 • (602) 553-0333
Additionally, many of our reports can be found in electronic format at:
www.azauditor.gov
DEBRA K. DAVENPORT, CPA
AUDITOR GENERAL
STATE OF ARIZONA
OFFICE OF THE
AUDITOR GENERAL
WILLIAM THOMSON
DEPUTY AUDITOR GENERAL
2910 NORTH 44th STREET • SUITE 410 • PHOENIX, ARIZONA 85018 • (602) 553-0333 • FAX (602) 553-0051
April 22, 2008
Members of the Arizona Legislature
The Honorable Janet Napolitano, Governor
The Honorable Tom Horne, State Superintendent of Public Instruction
Transmitted herewith is a report of the Auditor General, Baseline Study of Arizona’s English
Language Learner Programs and Data, Fiscal Year 2007. This special study addresses
certain audit requirements mandated by A.R.S. §15-756.12, and was conducted pursuant
to Arizona Revised Statutes §41-1279.03. I am also transmitting with this report a copy of
the Report Highlights to provide a quick summary for your convenience.
This report describes a sample of ELL programs in the State, providing a point of
comparison for future biennial audits, which will report on the effectiveness of the State’s
programs in achieving English proficiency. In addition, this report discusses the reliability
of data necessary to manage and audit the programs and recommends steps to improve
future data before the biennial audits begin.
As outlined in its response, the Arizona Department of Education agrees with all of the
findings and recommendations.
My staff and I will be pleased to discuss or clarify items in the report.
This report will be released to the public on April 23, 2008.
Sincerely,
Debbie Davenport
Auditor General
The Office of the Auditor General has conducted a baseline study of the current state
of English Language Learner (ELL) programs and available data. The report
describes the ELL programs and participants in a sample of 18 school districts and
charter schools, chosen to provide a cross-section of districts and charter schools
across the State. This report also provides a point of comparison for future biennial
audits, which are mandated by A.R.S. §15-756.12 to report on the effectiveness of
the State’s ELL programs in achieving English proficiency. In addition, this report
discusses the reliability of data necessary to manage and audit the programs and
recommends steps to improve future data before the biennial audits begin. To place
Arizona’s ELL programs in a national context, recent ELL data and trends from
across the nation are presented.
ELL students and programs in Arizona and in the U.S.
(see pages 1 through 9)
Nation-wide, ELL students have composed 10 percent of the total student enrollment
for kindergarten through 12th grade. In Arizona—one of five states with the highest
concentration of ELL students—the 138,449 ELL students composed approximately
14 percent of the State’s total enrollment in fiscal year 2007. Although ELL students
speak more than 400 different languages, Spanish is spoken by 80 percent of all ELL
students nationally and 81.2 percent in Arizona. In Arizona, as in the nation as a
whole, most of the ELL students are in elementary grades. The percentage of
Arizona’s ELL students diminishes consistently from kindergarten through 12th
grade. ELL students constitute over 15 percent of Arizona kindergartners and less
than 2 percent of its 12th graders.
Arizona’s approach to ELL standards and assessment differs from many other
states. Since 2004, Arizona’s districts and charter schools have relied on state-defined
proficiency standards. In contrast, many districts in other states also use
standards at the district, school, and even classroom level. For program entry, exit,
and monitoring, Arizona’s districts are required to use the Arizona English Language
Learner Assessment (AZELLA), a composite of speaking, listening, reading, and
writing scores. The assessments used in other states are not as comprehensive.
Office of the Auditor General
SUMMARY
page i
Ninety percent of districts surveyed in other states based entry decisions solely on
oral proficiency, and nearly 82 percent used oral proficiency as the sole basis for
exiting the program. Under the No Child Left Behind Act of 2001, the federal
government requires that all public school districts annually assess all students with
limited English proficiency, but neither defines proficiency nor specifies which tests
that states must use to measure it.
Arizona’s most recent ELL legislation, HB2064, was an outgrowth of more than a
decade of litigation and legislation, and further specifies the State’s ELL program
structure. The bill requires the ELL Task Force to develop state-wide program
models, ADE to monitor and report on the program’s success, and the Auditor
General to audit the models’ effectiveness on performance-based outcomes and
English proficiency. This study focused on current ELL programs operating at a
sample of Arizona school districts and charter schools and the data these schools
reported to ADE. The sample of 13 districts and 5 charter schools was designed to
reflect different types of programs based on district or charter school size, location,
number, and proportion of ELL population and percentage of ELL students
reclassified as English proficient.
Characteristics of sample ELL programs and participants
(see pages 11 through 19)
ELL programs in the 18 sample districts and charter schools differ greatly from each
other and from the structure they will have to employ in the future under the ELL Task
Force’s new Structured English Immersion (SEI) models. Adopted in September
2007, the new models specify state-wide ELL policy regarding instructional approach
and classroom practices. Implementation of these SEI models is expected beginning
in fiscal year 2009. In fiscal years 2007 and 2008, the sample districts and charter
schools operated ELL programs that varied by grade and proficiency levels and
across individual schools. Although programs at the sample districts and charter
schools were aligned with the new models’ requirements regarding assessment and
English-only classroom materials, their instructional approaches were significantly
different from the models’ future SEI requirement regarding English language
development (ELD).
The Task Force’s models require all new ELL students to receive daily 4-hour ELD
classes. In fiscal year 2008, only 3 of the 18 sample districts and charter schools were
operating classes that met this upcoming requirement, but only for a portion of their
ELL students. In 2007, the most recent year for which comprehensive data was
available, more than half of all ELL students in the sample districts and charter
schools attended programs that mainstreamed all ELL students, providing no hours
of ELD instruction in an SEI setting. Forty-two percent were in programs that provided
State of Arizona
page ii
up to 2 hours of daily ELD instruction. The remaining 6 percent provided more than
2 and up to 4 hours of daily ELD. Across sample districts and charters, program
officials identified the schedule and resource challenges to meeting the 4-hour daily
ELD requirement. For example, program officials at three rural districts stated that
they currently have difficulty filling regular teaching slots, and they do not know where
they would find four to six more teachers with the qualifications to teach SEI classes.
The new models are designed to bring ELL students to full proficiency in 1 year. For
the sampled districts and charter schools, about 7 percent of the approximately
8,700 ELL students became fully proficient in fiscal year 2007, and most of them had
been in the program for at least 2 years. Between fiscal years 2006 and 2007, nearly
two-thirds of the students remained at the same proficiency level or regressed, while
about one-third moved to a higher proficiency level. ELL students who were at the
lowest proficiency levels tended to make the most progress.
ELL data and data systems need attention (see pages 21
through 28)
A.R.S. §15-756.10 requires ADE to collect and maintain ELL data related to program
funding and monitoring. A.R.S. §15-756.12 requires the Auditor General to review
compliance with program requirements in district performance audits and biennially
report on the overall effectiveness of the State’s ELL program. Based on reviews of
the data submitted by sample districts and charter schools to ADE and of source
records, auditors found several errors affecting both the accuracy of funding ADE
provides to districts and charter schools and the measurement of ELL students’
progress.
To adequately fund, manage, and audit the effectiveness of Arizona’s ELL programs,
three main types of information are needed: the number of ELL students,
achievement outcomes, and time in the program. First, since state funding is
awarded on a per-student basis, ADE needs an accurate determination of the
number of ELL students in each district or charter school. However, auditors’ review
of sample files found that basic student identification information was not necessarily
present, or if present, was not necessarily correct. Further, 2 percent of the sample
ELL students who were funded had no assessment scores or had an assessment
score indicating that, prior to entering the program, the ELL student was already
proficient. Second, although accurate proficiency information is needed to monitor
and evaluate districts’ and charter schools’ success, for 6.5 percent of the students,
the classification of overall proficiency levels did not match their corresponding
AZELLA scores. Additionally, 4 percent who were reclassified as proficient actually
had scores that were below proficient or lacked valid assessments. Third, since
funding is limited to 2 years, ADE needs to know how long each ELL student has
Office of the Auditor General
page iii
been participating in the program. However, 6 of the 18 sample districts had ELL
students with questionable entry and exit dates. To prevent such errors, ADE needs
to work with district and charter schools to develop better ELL data submission and
review processes.
ADE does not have adequate procedures in place to identify these errors and either
correct them or minimize their impact. When data submitted by districts and charter
schools is inaccurate or incomplete, ADE’s own internal processes should be able to
identify the problems. Auditors found that ADE’s integrity checks and controls for
doing so are insufficient. Consequently, the errors and limitations in the data are
carried into the system. For example, based on schools’ erroneous data
submissions, ADE included English-proficient students in program-funding counts.
Further, an agency’s data system should include process controls to ensure that no
data is added, lost, or altered during processing. In August 2006, the Auditor
General’s performance audit of ADE’s information management recommended that
ADE add such controls. The presence of such controls could help users and ADE
prevent potential problems with student achievement and funding data, such as the
detection and correction of a massive state-wide recalculation and override of ELL
counts that occurred in July 2007. In this case, because of a processing error, over
20,000 ELL students were excluded from end-of-year funding counts. ADE eventually
corrected the $8 million error, but analysts struggled to identify its cause. Further, ADE
did not investigate prior years’ funded counts for similar processing exclusions.
Also, in order to better monitor implementation of the new SEI models and assess
program success, ADE could collect additional data that is currently available from
districts. Information about the number and qualifications of ELL teachers is not
currently collected and maintained by ADE, although ADE requires districts to
maintain ELL teacher certification and endorsement documentation for review during
monitoring visits. Information about the number of hours of ELD instruction could also
assist with monitoring efforts. Sample districts and charter schools provided auditors
with information on students’ hours of ELD instruction based on their proficiency and
grade levels.
State of Arizona
page iv
Office of the Auditor General
TABLE OF CONTENTS
continued
1
11
11
16
21
21
22
25
27
28
Introduction & Background
Chapter 1: Characteristics of Sample ELL Programs
and Participants
Current programs vary significantly
Few ELL students attained proficiency within 1 year
Chapter 2: ELL Data and Data Systems Need Attention
ELL data essential for funding, monitoring, and auditing
School data submitted to ADE contained errors
ADE’s controls and checks for data reliability are insufficient
ADE could expand types of data collected
Recommendations
Agency Response
page v
State of Arizona
TABLE OF CONTENTS
Tables:
1 Comparison of Arizona's ELL Program Requirements to Other
States' ELL Programs' Requirements
2 Sample District and Charter School Characteristics
Fiscal Year 2006
3 Annual Progress of ELL Students by Proficiency Levels Between
Fiscal Years 2006 and 2007
4 Elementary and High School ELL Students by Proficiency Level
Fiscal Year 2007
5 Fiscal Year 2006 Proficiency Level of ELL Students Reclassified as
Proficient in Fiscal Year 2007
6 Availability and Reliability of ELL Data at Sample Districts and
Charter Schools
Fiscal Year 2007
Figures:
1 Comparison of Percentages of Arizona to U.S. ELL students by
Grade-Level Group
2 Overview of ELL Process from Entry to Monitoring Period
3 Arizona's Statutory Provisions Related to ELL Program Structure,
Data and Reporting
4 Percentages of ELL Students Grouped by English Language
Development Hours
Fiscal Year 2007
3
8
17
18
19
23
2
4
6
14
concluded
page vi
A.R.S. §15-756.12 requires the Auditor General to conduct a biennial audit of the
effectiveness of school districts’ and charter schools’ English Language Learner
(ELL) programs. The statute calls for the audit to be based on outcome measures,
such as the percentage of ELL students who become proficient in
English each year. The biennial audits will begin in fiscal year 2010. As
a point of comparison for these future audits, this baseline study
establishes the current state of ELL programs and available data
across Arizona. It focuses on ELL programs in a sample of 18 school
districts and charter schools chosen to provide a cross-section of
districts and charter schools across the State. To place Arizona’s ELL
programs in a national context, this study also presents recent ELL data
and trends from across the nation.
The future structure of ELL programs in Arizona is based primarily on
Laws 2006, Ch. 4 (House Bill 2064), passed by the Legislature in 2006.
The bill specified that an ELL Task Force develop a state-wide model
for ELL instruction, and that ADE report ELL information collected from
districts as well as support and monitor their compliance with the
models. The models were adopted in September 2007. This baseline
study focuses on programs in place in 2006 and 2007, before the
models were adopted.
In order for ADE to manage the program and for the Auditor General to determine the
effectiveness of the models, the agencies need reliable information on performance
measures, such as ELL students’ proficiency levels and time in the program. Chapter
1 describes the sample districts’ and charters’ ELL programs and what current
performance measures say about ELL students’ proficiency and progress. Future
biennial audits will use this information for comparative purposes, as appropriate. To
help ensure that the information being collected and reported to ADE is as reliable
as possible, Chapter 2 of this report discusses the data necessary to manage and
audit the program; the data collection processes at ADE, districts, and charter
schools; the reliability of current data; and steps that need to be taken to improve
future data before the biennial audits begin.
Office of the Auditor General
INTRODUCTION
& BACKGROUND
page 1
English Language Learner
An individual who:
􀁺􇩩 is 3 to 21 years old
􀁺􇩩 is enrolled or preparing to enroll in an
elementary or secondary public school
􀁺􇩩 is not born in the U.S. and has a native
language other than English or whose
English proficiency is significantly
impacted by a language other than English
􀁺􇩨 has speaking, reading, writing, or
comprehension of English problems
sufficient to limit success in English
language classrooms, on state
assessments, and participating in society
Source: No Child Left Behind Act of 2001, Title IX, Part
A, section 9101.
English Language Learners in Arizona and in the United
States
Between 1991 and 2001, the number of ELL students in the United States more than
doubled. Nation-wide, there were more than 5 million ELL students in 2006,
composing 10 percent of the total student enrollment for kindergarten through 12th
grade. Arizona’s percentage is higher: in 2007, the 138,449 ELL students in Arizona’s
public schools composed approximately 14 percent of the State’s total enrollment.
According to a study sponsored by the U.S. Department of Education in 2000,
Arizona was one of five states with the highest concentration of ELL students, along
with California, New Mexico, Texas, and Alaska.
Although ELL students collectively speak
more than 400 different languages, Spanish
predominates. It is the language spoken by
80 percent of all ELL students nationally and
81.2 percent in Arizona. In Arizona, the
second most prevalent language is Navajo
(2.6 percent), while nation-wide, the second
most prevalent language is Vietnamese (2.4
percent).
In Arizona, and nation-wide, the majority of
ELL students are in elementary grades. Fifty-six
percent of Arizona’s ELL students are in
kindergarten through third grade, higher than
the 45 percent of students nation-wide in
these lower grades (see Figure 1). The
percentage of Arizona’s ELL students
diminishes consistently from kindergarten
through 12th grade. ELL students constitute
over 15 percent of Arizona kindergartners
and less than 2 percent of its12th graders.
Arizona’s approach to standards and assessment differs
from many other states
Under the No Child Left Behind Act of 2001, the federal government requires that all
public school districts provide an annual assessment test that measures the reading,
writing, and oral language skills of all students with limited English proficiency. The
act, however, neither defines proficiency nor specifies which tests that states must
use to measure it; these decisions are left to the states. Since 2004, Arizona districts
State of Arizona
page 2
45
56
36
32
19 12
0
10
20
30
40
50
60
70
80
90
100
U.S. AZ
Percentage of ELL Students
Kindergarten-3rd grade 4th-8th grade 9th-12th grade
Figure 1: Comparison of Percentages of Arizona to U.S. ELL Students
By Grade-Level Group
Source: Auditor General staff analysis of fiscal year 2007 ELL grade-level and program participation data
provided by the Arizona Department of Education for Arizona data and fiscal year 2001 information
from the National Clearinghouse for English Language Acquisition and Language Instruction
Education Programs’ Survey of States’ Limited English Proficient Students and Available
Educational Programs and Services 2000-2001 Report, October 2002 for U.S. data.
and charter schools have relied on state-defined proficiency standards. In contrast,
many districts in other states also use standards at the district, school, and even
classroom level (see Table 1 below, which compares Arizona to other states across
a number of ELL program components).
Office of the Auditor General
page 3
ELL Program Components
Arizona’s
Requirements
Other States’
Requirements
Level of government that
defines proficiency standards
(for districts in other states,
more than one category may
apply)
State State (61.6%)
District (54.3%)
School (23.8%)
Classroom (14.4%)
Other (0.7%)
Type of data used to identify
ELL students
Home language survey Home language survey (90.7%)
Type of data used to make
placement decisions
Oral, reading, and writing
proficiency tests in English—
Arizona English Language
Learner Assessment
(AZELLA)
Oral proficiency tests in English
(90.0%)
Frequency of ELL instruction
review
At least once per year Once each year (57.5%)
Twice or more per year (38.0%)
Other (4.5%)
Type of data used to make
reclassification decisions
AZELLA Oral proficiency tests in English
(81.8%)
Monitoring of reclassified
students
Yes Yes (66.3%)
No (33.7%)
Frequency of ELL
reclassification review
At least once per year Once each year (58.2%)
Twice per year (22.9%)
Other (18.9%)
Monitoring period of reclassified
students
2 years 2 years (44.3%)
1 year (31.2%)
Other (24.5%)
Type of data used to monitor
reclassified students
AZELLA Student grades (93.3%)
Table 1: Comparison of Arizona's ELL Program Requirements
To Other States’ ELL Programs' Requirements
Source: Arizona Revised Statutes Title 15, Chapter 7, Article 3.1, for Arizona's requirements and Hopstock, Paul J., Howard L. Fleischman, Michelle L. Pendzick, et
al. Descriptive Study of Services to LEP Students and LEP Students with Disabilities: Volume I Research Report. September 2003, Appendix B for other states'
ELL programs' requirements.
Figure 2 below shows a typical process for ELL program entry and exit in Arizona and
in many districts across the U.S. After students are identified as having a primary
home language other than English, they are assessed for English proficiency and, if
they test below proficiency, they are placed in an ELL program. When the annual
proficiency assessment shows that they meet proficiency standards, they exit the
program, are reclassified as fluent English proficient, and are monitored for a period
of time.
State of Arizona
page 4
Is the student as-sessed
as being pro-ficient
in English?
The student is monitored
for a period of time.
Is the studen t assessed
as NOT proficient in
English?
Does the stu dent have a
home language other
than English?
The student participates
in an ELL program.
The studen t does not
participate in an ELL
program.
Yes
Yes
Yes
No
No
Figure 2: Overview of ELL Process from Entry to Monitoring Period
Source: Arizona Revised Statutes Title 15, Chapter 7, Article 3.1 and Auditor General Staff analysis of Hopstock, et. al. Descriptive Study of Services to LEP students
and LEP Students with Disabilities: Volume 1. September 2003.
Arizona, like most states, uses a home language survey to
identify students who may be ELL students. Once students are
identified, Arizona public school districts and charter schools
currently are required to use the Arizona English Language
Learner Assessment (AZELLA) for program entry, exit, and
monitoring. The AZELLA recognizes five levels of proficiency,
ranging from understanding no English at all to fully proficient
(see textbox). Assessment under the AZELLA is a composite of
speaking, listening, reading, and writing scores. The
instruments used in other states are not as comprehensive.
Ninety percent of districts surveyed in other states based entry
decisions solely on oral proficiency, and nearly 82 percent used
oral proficiency as the sole basis for exiting the program.
After exiting the ELL program as proficient, all Arizona students
are monitored for 2 years, again using the AZELLA. By contrast,
one-third of the districts surveyed in other states did not monitor
students who had been reclassified as proficient. Of the districts
that monitor, only 44 percent reported that they did so for 2
years. Further, 93 percent of the districts relied on student
grades rather than a testing instrument to monitor proficiency.
ELL programs, data, and reporting shaped by HB2064
During fiscal years 2006 and 2007, Arizona districts and charter schools were
operating three main types of ELL programs:
􀁺􇩓 Structured English Immersion (SEI)— In SEI programs, most or all instruction is
in English, and teachers use an instructional curriculum designed for children
who are learning the language. The goal is to help students learn the language
and become English proficient in the shortest amount of time.
􀁺􇩂 Bilingual—Bilingual programs use native language instruction to teach subject
matter with the goal of gradually shifting instruction to English over time. Since
2001 and as mandated by Proposition 203, bilingual education in Arizona
requires a waiver.
􀁺􇩍 Mainstream—These programs place students in regular classrooms with
English-fluent students.
Arizona’s most recent ELL legislation, Laws 2006, Ch. 4 (HB2064), further specifies
the type of SEI instruction that districts and charter schools will need to adopt in the
future. This bill was an outgrowth of more than a decade of litigation and legislation,
Office of the Auditor General
page 5
Arizona English Language Learner
Assessment
Levels of English language proficiency:
-E Student does not understand enough
language to perform in English.
Student understands and can speak a few
isolated English words.
Student may understand slower speech, and
speak, read, and write simple words and phrases, but
often makes mistakes.
Student can understand familiar topics
and is somewhat fluent in English, but has difficulty
with academic conversations.
Student can read and understand texts
and conversations at a normal speed, and can speak
and write fluently with minor errors.
Source: Arizona Department of Education.
discussed in more detail in an April 2007 Auditor General Report, Financing Arizona’s
English Language Learner Programs, Fiscal Years 2002 through 2006. As seen in
Figure 3 below, the bill specifies that the ELL Task Force will develop state-wide
program models, ADE will monitor and report on program success, and the Auditor
General will audit the models’ effectiveness on performance-based outcomes and
English proficiency.
State of Arizona
page 6
Arizona Department of Education
Arizona ELL Task Force—
• Develop and adopt research-based, cost-effective Structured English
Immersion (SEI) models.
• Annually re-evaluate the models, and revise if necessary.
Office of English Language Acquisitiion Services
• Require each district and charter school to report annually the number and
types of ELL participants.
• Provide technical assistance and monitor programs for implementation of
SEI models.
• Report the mobility of ELL students within and between districts and charter
schools.
• Annually present a detailed summary of ELL programs and funding to the
State Board of Education.
• Report information relating to the demonstrated success of schools and
school districts at achieving English proficiency for ELL students.
Office of the Auditor General
• Conduct a biennial audit of the overall effectiveness of ELL programs based
on outcome measures and English proficiency with consideration for
student mobility.
• Review compliance with ELL program requirements in performance audits
of randomly selected districts.
Figure 3: Arizona’s Statutory Provisions Related to ELL Program Structure,
Data and Reporting
Laws 2006, Ch. 4 (House Bill 2064) effective September 2006
Source: Auditor General staff analysis of Laws 2006, Ch. 4 (HB 2064).
In September 2007, the ELL Task Force adopted its research-based models for SEI
classes, which are required to be used in all Arizona schools unless, on a case-by-case
basis, the Task Force approves an alternative approach proposed by a district
or charter school. The models, discussed more fully in Chapter 1, specify state-wide
ELL policy and provide guidance for program structure and classroom practices.
In order to comply with HB2064 mandates, ADE and the Office of the Auditor General
must analyze ELL information that is collected from districts and charter schools and
maintained by ADE. Auditors identified several research questions that need to be
addressed in reviewing compliance with ELL program models and determining the
program’s effectiveness in achieving English proficiency. These questions, which
relate both to how the programs are structured and what results they achieve, include
the following:
Compliance—
􀁺􇩈 How many ELL students are served and are they concentrated in particular
schools within a district? How are they grouped and placed in the program?
􀁺􇩈 How is the program being implemented? How many hours of English
language instruction do ELL students receive?
Effectiveness—
􀁺􇩄 Does the program have a high percentage of ELL students who become
reclassified as English proficient?
􀁺􇩈 How long does it take a student to become English proficient, on average? Is
the average time in the program significantly different for students entering
with different proficiency levels—Pre-Emergent, Emergent, Basic,
Intermediate?
􀁺􇩉 Is student mobility—the extent to which students transfer between schools—
associated with progress in English proficiency?
􀁺􇩗 Which students typically re-enter the program after being reclassified as
English proficient?
Performance audits of randomly selected districts will report on how the models are
being implemented, and biennially, the Office of the Auditor General will audit the
program’s overall effectiveness.
Office of the Auditor General
page 7
Scope and Methodology
This study focused on current ELL programs operating at a sample of Arizona school
districts and charter schools. The sample of 13 districts and 5 charter schools was
designed to reflect different types of programs, but is not statistically representative
of all Arizona ELL programs. As seen in Table 2, the sample selection was based on
district or charter school size, location, number, and proportion of ELL population and
percentage of ELL students reclassified to “proficient.” Based on the distribution of
fiscal year 2006 reclassification data, auditors selected 9 districts with relatively high
reclassification rates— above 33 percent—and 9 with relatively low rates—below 10
percent. The sample included 4 large districts, 7 medium districts, and 7 small
districts and charters. Of the 18 districts and charters, 9 were in rural and 9 were in
urban areas. Five sample programs had more than 25 percent of their total student
population consisting of ELL students.
State of Arizona
ELL
Students
Reclassified
Students
Reclassification Rate
School
Size Location Number Percent Number Percent District or Charter School Name
High
Large
Rural 2,474 41% 1,039 42% Nogales Unified School District
Urban 3,615 27 1,235 34 Glendale Elementary School District
Medium
Rural 148 5 119 80 Liberty Elementary School District
Rural 50 4 23 46 Indian Oasis-Baboquivari Unified School District
Urban 100 16 65 65 Southgate Academy, Inc.
Urban 246 16 82 33 Sedona-Oak Creek Joint Unified School District
Small
Rural 36 10 12 33 Antelope Union High School District
Urban 104 47 54 52 Pan-American Elementary Charter School
Urban 34 9 10 29 Clarkdale-Jerome Elementary School District1
Low
Large Rural 75 1 0 0 Vail Unified School District
Urban 281 4 1 0 Tolleson Union High School District
Medium
Rural 277 35 0 0 Riverside Elementary School District
Urban 139 19 8 6 West Phoenix Public Charter High School
Urban 382 8 7 2 Florence Unified School District
Small
Rural 10 13 1 10 Chester-Newton Charter and Montessori School
Rural 7 4 0 0 Sedona Charter School, Inc.
Rural 105 28 2 2 Wellton Elementary School District
Urban 57 11 1 2 Ajo Unified School District
page 8
Table 2: Sample District and Charter School Characteristics
Fiscal Year 2006
1 The district’s reclassification rate did not meet the threshold for the high reclassification rate group, but had the highest rate of small, urban public
school districts and was included as an example of that group.
Source: Auditor General staff analysis of fiscal year 2007 School District Annual Financial Reports, average daily membership information provided by the Arizona Department of
Education, and summary accounting data provided by individual school districts.
In conducting this study, auditors used a variety of methods, including conducting
site visits to observe all sample programs, and analyzing various records and data
collected and compiled by ADE and information obtained directly from various
school districts and charters. Specifically:
􀁺􇩔 To provide a background of ELL legislation, Arizona Fluency Standards, AZELLA
components, and ELL Task Force SEI models, auditors reviewed HB 2064,
current Task Force SEI models, and documents provided by ADE.
􀁺􇩔 To provide a national context for Arizona’s ELL program structure, population,
and data collection processes, auditors reviewed national ELL studies and
interviewed program staff in other states about their ELL legislation and related
data systems.
􀁺􇩔 To provide an understanding of how each of the programs in the sample
operated, auditors interviewed program staff, visited program sites, performed
file reviews, and observed classrooms with ELL students. Program staff
provided detailed schedules for fiscal year 2007 ELL classes and more general
information on fiscal year 2008 schedules.
􀁺􇩔 To analyze sample ELL demographic, program participation, and outcome data,
auditors used student-level data collected and reports prepared by ADE. School
districts and charter schools put specific student data into ADE’s Student
Accountability Information System (SAIS), and ADE extracts and reports ELL-related
information from SAIS. As a special study, the information was not
subjected to all the tests and confirmations that would be performed in an audit.
However, auditors validated some of this district- and charter-reported data,
including assessment scores and primary home language, by comparing it to
the schools’ original supporting records. Auditors also performed
reasonableness tests to SAIS data for sample districts and charter schools.
􀁺􇩔 To assess the reliability of SAIS processing of ELL data, auditors reviewed SAIS
documentation related to the implementation of statutory program requirements
and process controls, interviewed ADE officials, and analyzed the data for
logical errors.
The Auditor General and staff express their appreciation to the Superintendent of
Public Instruction, the staff of the Arizona Department of Education, and the staffs of
the Arizona public school districts and charter schools for their cooperation and
assistance during this study.
Office of the Auditor General
page 9
State of Arizona
page 10
Characteristics of sample ELL programs and
participants
ELL programs in the 18 sample districts and charter schools differ greatly from each
other and from the SEI structure they will have to employ in the future under the
models adopted by the ELL Task Force. The Task Force’s models require all new ELL
students to receive daily 4-hour English language development (ELD) classes. In
fiscal year 2008, only 3 of the 18 sample districts and charter schools were operating
classes that met this upcoming requirement, but only for a portion of their ELL
students. In 2007, the most recent year for which comprehensive data was available,
more than half of all ELL students in the sample districts and charter schools
attended programs that mainstreamed all ELL students during regular classroom
hours. The new models are designed to bring ELL students to full proficiency in 1
year. For the sampled districts and charter schools, about 7 percent of the
approximately 8,700 ELL students became fully proficient in fiscal year 2007, and
most of them had been in the program for at least 2 years. Between fiscal years 2006
and 2007, nearly two-thirds of the students remained at the same proficiency level or
regressed, while about one-third moved to a higher proficiency level. ELL students
who were at the lowest proficiency levels tended to make the most progress.
Current programs vary significantly
The Task Force’s new models for Structured English Immersion (SEI) classes,
adopted in September 2007, specify a state-wide ELL policy regarding providing
instruction to ELL students. These models are expected to be implemented
beginning in fiscal year 2009. In fiscal years 2007 and 2008, the sample districts and
charter schools operated ELL programs that varied by grade and proficiency levels
and across individual schools. Although the new models call for ELL students to
receive 4 hours of ELD classes a day in SEI settings, the majority of ELL students in
the sample programs were being taught in programs that mainstreamed ELL
Office of the Auditor General
page 11
CHAPTER 1
students with other students during regular classroom hours. Most districts cited
scheduling and resource challenges in moving to the new models.
New state SEI models—The ELL Task Force’s research-based models for SEI
classes, which all Arizona schools are required to use unless the Task Force
approves an alternative approach, specify state-wide ELL policy and provide
guidance for program structure and classroom practices.
􀁺􇩐 Policy—ELD is to be taught for a minimum of 4 hours a day for the first year a
student is classified as an ELL. All new ELL students who test below
“proficient” must receive the 4 daily ELD hours, regardless of grade. In their
second year, middle and high school ELL students testing at the Intermediate
level may be excused from 1 to 2 hours of ELD instruction. Programs should
teach English using only English materials and instruction in an SEI setting
with the goal for ELL students to become proficient in 1 year.
􀁺􇩓 Structure—
o Content—SEI class content
is a minimum of 4 hours of
ELD, a type of instruction
designed for students who
are in the process of learning
English. Unlike math and
science classes, ELD
emphasizes concepts such
as semantics, syntax, word
structure, and pronunciation.
o Placement—ELL students are placed into SEI classes based on their
English proficiency on the AZELLA and their grade level.
o Grouping—Depending on the number of ELL students in each grade and
proficiency level, programs should group students with students of
similar proficiency levels. As long as the 4-hour ELD requirement is met,
districts and charter schools can adjust their ELD classes based on the
school size and grade, number and proportion of ELL students attending
each school.
State of Arizona
page 12
English Language Development
The teaching of English language skills
to ELL students with content
emphasizing the English language itself;
focusing on pronunciation, structure,
syntax, vocabulary, and semantics.
Source: SEI Models, AZ ELL Task Force, September 15, 2007.
􀁺􇩃 Classroom practices—The teaching materials used in an ELD class may
reflect content from a variety of academic disciplines, such as math or
science, but must be only in English, be appropriate for the students’ level of
proficiency, and be chosen based on effectiveness in promoting specific
English language objectives.
Substantial variety in existing programs—Although the state SEI models
may result in a more uniform SEI approach across the State, the current programs
in the 18 sample districts and charter schools reflected a variety of mainstream and
SEI approaches. None of the programs offered bilingual education. However,
according to district officials, one of the charters that used a mainstream approach
obtained bilingual waivers for all new ELL students because teachers were allowed
to speak Spanish for an initial period until the ELL students were integrated into
mainstream classes.
Within individual districts and charter schools, programs also reflected
considerable variation across grade levels, levels of proficiency, or individual
schools.
􀁺􇩇 Grade level—In five of the sampled districts and charter schools, the
instructional setting varied by grade level. Typically, these programs provided
mainstream instruction in the elementary grades, while ELD classes in SEI
settings were offered at the high school grade levels.
􀁺􇩐 Proficiency level—Two districts provided SEI classes across grade levels, but
not for ELL students who tested at the intermediate level. These students were
mainstreamed instead. A program official in one district said the district lacked
the space to provide separate classes for all ELL students and that
intermediate students were better able to adjust to the mainstream setting
than students with lower proficiency levels.
􀁺􇩓 School—Four districts varied the programs among their schools. For
example, officials in one district said they allow school principals to establish
their own programs to better meet their specific ELL student population’s
needs as long as they are within the state requirements for ELL instruction.
Similarly, at two other districts, some schools provided ELL-only classes, while
others mainstreamed ELL students.
In a few programmatic aspects, the sample schools were more aligned with the
upcoming SEI models. First, all of the school districts and charter schools reported
that they used the AZELLA for program placement and exit. Second, all schools
that used an SEI approach used proficiency level as a grouping criteria. Third, in
all their classroom observations, auditors found instruction and materials were only
in English, in keeping with the SEI models’ specifications for classroom practices.
Office of the Auditor General
page 13
Most existing programs differed substantially from new state
requirements—The state SEI models’ policy specifies that ELL students
should receive 4 daily hours of ELD instruction in SEI settings. Most of the 18
sampled programs used a much different approach in fiscal years 2007 and 2008.
In fiscal year 2007, 7 of the 18 sample districts and charters offered programs
entirely in mainstream settings and provided no ELD hours to ELL students. As
seen in Figure 4, 53 percent of the approximately 8,700 sample ELL students were
in these seven programs. The next largest group, 38 percent, was in four programs
that provided between 1 and 2 hours of daily ELD instruction. Two programs
provided between 2 and 4 hours of daily ELD in fiscal year 2007, with only one of
them meeting the upcoming 4-hour requirement.
State of Arizona
page 14
1 The number of ELL students with 0 ELD hours may be higher because auditors could only
determine whether ELD was offered to some students at the district or charter school. Other
ELL students at those same districts or charter schools may have received no ELD because
their grade- or proficiency-level or school was mainstreamed.
Source: Auditor General staff analysis of fiscal year 2007 ELL schedules and survey responses provided by district and
charter school program officials, interviews with program staff, and fiscal year 2007 ELL participation
information provided by the Arizona Department of Education.
Figure 4: Percentages of ELL Students Grouped by
English Language Development Hours1
Fiscal Year 2007
Up to 1 hour/day
4%
Between 2 and 4 hours/day
5%
Between 1 and 2 hours/day
38%
0 hours
53%
At four of the seven mainstreaming programs, program staff
identified their ELL programs as SEI, but auditors determined that
these programs did not provide their ELL students with separate
ELD instruction during regular school hours. In some of these
districts and charters, the ELL students were instructed in
mainstream settings by teachers using approaches, such as
Sheltered Instruction Observation Protocol (S.I.O.P.), which uses
demonstrations, visual aids, and dialogue to enhance language
acquisition. In visits to mainstream classrooms, auditors
observed significant variation in the implementation of this
language-acquisition instructional approach. For example, some
teachers were very interactive and aware of the ELL students’
participation in class. In other cases, the teacher could not
identify the ELL students and/or used a traditional lecture
instructional approach.
Although some shift to more ELD instruction was discernible in
the programs in place for fiscal year 2008, most of the sampled
districts and charter schools were still far from having programs
that would meet the upcoming requirement. Auditors reviewed
the sample districts’ and charter schools’ programs for the current year—fiscal
year 2008—to determine what changes, if any, had occurred from fiscal year 2007,
the last year for which comprehensive data was available. Only 3 of the 18 sample
districts and charter schools offered 4 hours of daily ELD to any of their ELL
students. One was a large high school district with a small percentage of ELL
students. This district’s program, offered across all district schools, currently
teaches 4 hours of ELD to all ELL students, except those who test at the
intermediate level. The other two districts’ programs are in the process of
implementing Office of Civil Rights (OCR) agreements related to their ELL
programs. The OCR typically enters into an agreement after substantiating a
discrimination claim against the district. The OCR allows schools under OCR
agreements to provide instruction in SEI settings as long as the ELL program
placement, assessment, and exit policies are nondiscriminatory. At one district, in
efforts to redress the national origins claims made to OCR and in anticipation of
the State’s 4-hour ELD requirement, the district’s plan specifies that ELL students
receive 1 to 4 hours per day of ELD instruction depending on their proficiency level
and time in the program. At the other district under an OCR agreement, the 4-hour
ELD is available only at one of the district’s eight schools.
Office of the Auditor General
page 15
Office of Civil Rights Agreements
and Structured English Immersion
“Valid educational reasons may exist to
assign a student to a class with a
disproportionate number of minority or
nonminority students when a class
provides specially designated instruction
to enable ELL students to acquire English
language skills. These assignments are
allowed only when appropriate and
nondiscriminatory evaluation, placement,
and exiting criteria procedures are
followed and the classes are designed to
move students to regular classes within a
reasonable amount of time.”
Source: Office for Civil Rights, Title IV assignment.
Officials cite schedule and resource challenges in providing 4-
hour ELD instruction—Across sample districts and charters, program
officials identified the following schedule and resource challenges to meeting the
4-hour daily ELD requirement:
􀁺􇩋 Kindergarten students—The Task Force SEI model requires all ELL students,
including kindergarten students, to receive 4 hours of daily ELD in the SEI
setting. According to one program official, the challenge of finding additional
space for separate kindergarten classrooms for ELL students seemed
unnecessary because instructional approaches to ELL and non-ELL
kindergartners are similar.
􀁺􇩓 Staffing and space—Program officials from five of the sample districts and
charters stated that they would not have the staffing or space to meet the 4-
hour requirement. Three rural districts stated that they currently have difficulty
filling regular teaching slots, and they do not know where they would find four
to six more teachers with the qualifications to teach SEI classes. Another
sampled district cited lack of physical space available for the smaller “pull-out”
SEI classes. During the classroom visits, auditors observed SEI classroom
sizes between 3-20 students.
􀁺􇩅 ELL students spread across grades and proficiency levels—Grouping ELL
students’ SEI classes by proficiency level is complicated when a district or
charter has ELL students with various proficiency levels spread across grade
levels and/or schools. For example, a sample elementary school provided
ELD classes to new ELL students in grades 3 through 8 who tested below
intermediate. For these 12 students, in addition to finding ELD teaching
materials appropriate across their different age groups, the ELL coordinator
had to integrate the ELD classes with each ELL student’s grade-level
mainstream classes, such as math, science, and physical education. Some of
these ELL students switched between the ELD and mainstream classrooms
as often as six times in a day.
Few ELL students attained proficiency within 1 year
Between fiscal years 2006 and 2007, 63 percent of the sample ELL students
remained at the same proficiency level or actually regressed. Those who did advance
tended to be at the lowest proficiency levels, such as pre-emergent, rather than at
the intermediate level. Seven percent of the approximately 8,700 sample ELL
students in the sample programs in fiscal year 2007 became fully proficient by the
end of that fiscal year. Most of them had tested as intermediate the year before, and
most had been in the program for at least 2 years.
State of Arizona
page 16
Majority of sample ELL students’ proficiency remained the same or
regressed—Between fiscal years 2006 and 2007, 55 percent of the ELL
students in the sample districts and charters remained at the same proficiency
level, 8 percent fell back, and 37 percent moved forward one or more proficiency
levels. According to ADE program officials, students testing at the pre-emergent
and emergent levels
typically progress more
rapidly than students
testing at the basic and
Intermediate levels. These
different rates of progress
in proficiency were evident
in the sample districts and
charters. For example, as
seen in Table 3, 36 percent
of ELL students with pre-emergent
language skills
advanced three proficiency
levels in 1 year to achieve
the intermediate level, while
51 percent with emergent
skills advanced two
proficiency levels to
achieve the intermediate
level in 1 year. In contrast, two-thirds of the ELL students who tested as
intermediate in fiscal year 2006 remained at that proficiency level in fiscal year 2007
assessments. Further, almost one-third of the sample ELL students who tested as
proficient in 2006 regressed into the program the next year.
Half of sample ELL students at intermediate proficiency—In fiscal year
2007, approximately one-half of the ELL students in sample school districts and
charter schools tested at the intermediate proficiency level. According to officials,
since students at the intermediate level tend to make slower progress to the next
level than their counterparts at lower proficiency levels, the high percentage of
students testing at the intermediate level may help explain the overall lack of
proficiency growth in the sample. As seen in Table 4 (see page 18), 55 percent of
the approximately 8,700 ELL students tested as intermediate prior to participating
in fiscal year 2007 programs. Thirty-one percent tested as basic proficiency, with
the remaining 14 percent testing as pre-emergent or emergent. In all, more than
7,600 of the ELL students in the sample (87 percent) attended elementary grades
kindergarten through 8, while 13 percent attended grades 9 through 12. Sixty-seven
percent of high school students tested at the intermediate level, compared
to 53 percent of the elementary ELL students.
Office of the Auditor General
page 17
Proficiency Level
Percentage of ELL students
changing or maintaining proficiency level
Regressed Maintained Up1 Up2 Up3 Up4
Pre-Emergent 4% 5% 52% 36% 3%
Emergent 1% 3 43 51 2
Basic 2 36 53 9
Intermediate 8 67 25
Proficient 31 69
Table 3: Annual Progress of ELL Students by Proficiency
Levels Between Fiscal Years 2006 and 2007
Source: Auditor General staff analysis of ELL students’ participation and proficiency data for the sample districts
and charter schools for fiscal years 2006 and 2007 provided by the Arizona Department of Education.
Seven percent of sample ELL students reclassified as proficient,
most after 2 or more years in program—In fiscal year 2007, 596, or 7
percent, of the sampled districts’ and charters’ ELL students were tested and
reclassified as proficient. According to ADE, the State’s average reclassification
rate in fiscal year 2006 was 14.4 percent.1 It is not clear whether the different rates
are due to uniqueness of sample districts and charters, or to errors in state-wide
data processing, as discussed further in Chapter 2. In the sample districts and
charters, approximately three out of four (74 percent) of those reclassified in fiscal
year 2007 were in an ELL program for at least 2 years.2 Three percent were in a
program for 1 year or less. As Table 5 (see page 19) shows, approximately 89
percent of those reclassified as proficient in fiscal year 2007 had a prior proficiency
level of intermediate, and over 8 percent had a proficiency level of basic. About 3
percent, or 17 students, had tested at the pre-emergent or emergent level in fiscal
year 2006 before testing as proficient in fiscal year 2007.
1 “Demographic Characteristics and Assessment Results of ELL students in Arizona,” presented by ADE, ELL Task Force
Meeting, Phoenix, AZ, January 2007.
2 For students in an ELL programs for 2 or more years, auditors could not determine how long they were actually in the
program because reliable data prior to fiscal year 2005 is unavailable. Fiscal year 2005 was the first year that districts and
charter schools were required to use a state-wide ELL assessment to support program participation. It was also the first
year that ADE added processing controls related to ADE data.
State of Arizona
page 18
Proficiency Level
ELL Students
Elementary High School Total
Count Percent Count Percent Count Percent
Pre-Emergent 732 4% 39 4% 771 9%
Emergent 435 6 48 4 483 5
Basic 2,402 32 282 25 2684 31
Intermediate 4,021 53 761 67 4782 55
Total 7,590 1,130 8,720
Table 4: Elementary and High School ELL Students by Proficiency
Level1
Fiscal Year 2007
1 Proficiency levels for fiscal year 2007 participants were based on most recent, valid assessment
scores from fiscal years 2006 or 2007.
Source: Auditor General staff analysis of ELL students' participation and proficiency data for the sample districts and charter
schools for fiscal years 2006 and 2007 provided by the Arizona Department of Education.
Office of the Auditor General
page 19
2006 Proficiency Level
Reclassified as
Proficient ELL Students
Number Percent
Pre-Emergent 13 2.2%
Emergent 4 0.7
Basic 51 8.6
Intermediate 528 88.5
Total 596 100.0%
Table 5: Fiscal Year 2006 Proficiency Level of ELL Students
Reclassified as Proficient in Fiscal Year 2007
Source: Auditor General staff analysis of ELL students’ participation and proficiency data for the sample
districts and charter schools for fiscal years 2006 and 2007 provided by the Arizona Department
of Education.
State of Arizona
page 20
ELL data and data systems need attention
ADE needs to take action to ensure that accurate and meaningful data is available
so that it can adequately fund, manage, and audit the effectiveness of Arizona’s ELL
programs. Although ADE collects a wide variety of data from districts about their ELL
programs, auditors’ review of this data showed that it contains errors in all but one of
the 11 key fields for which data is currently available. Further, ADE does not have
adequate procedures in place to identify these errors and either correct them or
minimize their impact. These errors affect both the accuracy of funding ADE provides
to districts and charter schools and the measurement of ELL students’ progress.
ADE’s procedures would also be enhanced by collecting additional data that is
currently available from districts.
ELL data essential for funding, monitoring, and auditing
A.R.S. §15-756.10 requires ADE to collect and maintain ELL data related to program
funding and monitoring. Accurate and complete data is essential for several reasons:
􀁺􇩐 Program funding—Most ELL funding is based on the number of ELL students
served.1 ELL per-pupil funding, commonly called the “ELL B-Weight” in
reference to the school funding formula, is a calculated percentage of a
student’s base-level funding. Accordingly, the correct identification of the
number of ELL students is essential for determining the correct percentage for
districts and charter schools.
􀁺􇩍 Monitoring—ADE is responsible for monitoring program compliance by districts
and charter schools, for funding programs, and for reporting the extent to which
ELL students achieve English proficiency. To carry out this responsibility, ADE
needs accurate and complete data on how long students remain in the program
and the level of proficiency they have achieved. For example, ADE must annually
1 For further discussion, see the Office of the Auditor General’s report, Financing Arizona’s English Language Learner
Programs, Fiscal Years 2002 through 2006, April 2007.
Office of the Auditor General
page 21
CHAPTER 2
present to the State Board of Education detailed summaries of the program’s
success in achieving proficiency and of program funding.
􀁺􇩁 Auditability—A.R.S §15-756.12 requires the Auditor General to review
compliance with program requirements in district performance audits and
biennially report on the overall effectiveness of the State’s ELL program. In
performance audits of randomly selected districts, auditors will review programs
for compliance with ELL program requirements. In biennial audits of the State’s
programs, auditors will report on the ELL programs’ overall effectiveness based
on outcome measures and English proficiency, considering student mobility. If
the data itself is unreliable, neither the Auditor General nor any other entity can
attest that the analysis presents an accurate picture of what the program is
accomplishing.
To fund, monitor, and audit ELL programs, three main types of information are
needed:
􀁺􇩎 Number of ELL students—Since state funding is awarded on a per-student
basis, ADE needs an accurate determination of the number of ELL students in
each district or charter school.
􀁺􇩁 Achievement outcome—To monitor and evaluate districts’ and charter schools’
success in bringing ELL students to proficiency in English, ADE needs
information about each student’s level of proficiency as they enter the program
and the changes in this proficiency as they move through their instruction.
􀁺􇩔 Time in the program—Statute provides districts and charter schools with 2 years
of ELL funding. Since funding is limited to 2 years, ADE needs to know how long
each ELL student has been participating in the program. Because families with
ELL students may relocate, ADE needs to be able to track ELL students’ time in
the program for all districts and charter schools as well.
School data submitted to ADE contained errors
Auditors found errors in most of the data fields that districts and charter schools
submitted to ADE’s Student Accountability Information System (SAIS). Auditors
conducted two main types of reviews—a detailed analysis of 493 ELL files from the
18 districts and charter schools in the sample, and a more general analysis of
records for all 10,054 ELL students who attended the sample districts and charter
schools in fiscal year 2007.1 These reviews showed that although districts and
charter schools submit many types of information that can be useful in administering
and evaluating the program, most of the available information was found to be
unreliable. Auditors identified 15 ELL-related data fields that would be useful in
administering the program or auditing its effectiveness (see Table 6 on page 23). The
1 When the 10,054 sample ELL students’ memberships are summarized for funding purposes and adjusted by auditors
for errors, they equate to the approximate 8,700 full-time equivalent ELL students cited in Chapter 1.
State of Arizona
page 22
majority of these data types—11 of the 15—are currently available in ADE’s SAIS
system. However, only one of the 11 available data types—grade level—can
currently be categorized as reliable. The errors and processing issues of the other ten
data types are discussed in subsequent sections.
Unreliable data limit ADE and the Auditor General from providing accurate basic
information and answering critical research questions. For example:
􀁺􇩗 What is the primary home language spoken by the ELL student?
Current SAIS data suggests that for 11 percent of the ELL students English is
the primary language spoken at home. This is probably higher than the actual
Office of the Auditor General
page 23
Type of data
Is it available
from ADE? Is it reliable?
Data on individual students
Grade level    
Home language  
Program entry date  
Program exit date  
AZELLA total composite score  
Overall proficiency level  
Overall assessment result  
Reclassification as fluent English
proficient (FEP)  
Data on program in general
Number of ELL students  
Number of SEI-endorsed teachers n/a
Number of teachers with ESL or
bilingual certifications n/a
SEI teachers’ average years of
experience n/a
Hours of ELD instruction n/a
Number of ELL students reclassified
as FEP  
Average time in program until FEP  
Table 6: Availability and Reliability of ELL Data
at Sample Districts and Charter Schools
Fiscal Year 2007
Source: Auditor General staff analysis of ELL students’ demographic, program participation, and proficiency data for the sample districts and
charter schools for fiscal year 2007 provided by the Arizona Department of Education and comparison of a sample of this data to
supporting records provided by sample districts and charter schools.
percentage because students are assessed for program participation when
the language spoken at home is not English.
􀁺􇩈 How long does an ELL student typically participate in the program before
becoming proficient?
Auditors found errors in the program exit information in 6 of 18 sample
programs and errors in the proficiency levels for all of the programs.
􀁺􇩗 What is the average percentage of ELL students reclassified as proficient in a
given year?
Four percent of sample ELL students who were reclassified as proficient were
actually assessed below proficient or lacked valid assessments.
The specific problems with reliability are discussed in the sections that follow.
Errors in student identification—Basic identification information about each
ELL student includes such data as a unique identification number and the
student’s home language. Such information is essential for tracking students’
eligibility and participation across the years. Auditors’ review of 493 sample files
found such information was not necessarily present, or if present, was not
necessarily correct.
􀁺􇩉 Identification number—Within SAIS, each student has a unique identification
number, called a SAIS ID. To monitor students over time, SAIS IDs should be
consistent across years and assigned to only one student. Before requesting
a SAIS ID for a new student, school officials should determine whether a SAIS
ID has already been assigned to the student by another school. Within the
sample of 493 files, auditors identified instances in which an ELL student
received a new SAIS ID in a subsequent fiscal year, even though the
membership district remained the same, and the same SAIS ID was assigned
to one ELL student in fiscal year 2006 and a different ELL student in 2007.
􀁺􇩈 Home language—Twenty percent of the ELL students in the sample lacked
appropriate or current documentation of their home language. In SAIS data,
11 percent of the ELL students at the sample districts and charter schools
have English as a primary home language, which would tend to call into
question whether the student should be classified as an ELL student.
According to program officials at one sample charter school, their student
management information software has a default value of English for primary
home language, and they do not enter other values, even when ELL students
indicate Spanish as their home language on the state form.
Errors in student proficiency and participation data—Accurate
information about an ELL student’s English proficiency is critical for determining
appropriate funded-participant counts, achievement outcomes, and time in the
State of Arizona
page 24
program. In Arizona, students receive a score on the AZELLA and a resulting
proficiency level. For students scoring below proficient, officials also enter
information about program participation. Auditors identified errors in data related
to the assessment of students’ English proficiency and their program participation.
􀁺􇩔 Total composite AZELLA score—In 14 of the 493 sample files (3 percent),
auditors found that the information in a student’s file could not support the
information that had been entered into SAIS. In fiscal year 2007, the district
and charter schools entered AZELLA scores manually into their data systems
and reported them to ADE.
􀁺􇩏 Overall proficiency level—For each total composite AZELLA score, school
employees classified the student in one of the five corresponding overall
proficiency levels ranging from pre-emergent to proficient. Auditors found that
for 6.5 percent of the students, or 658 of the 10,054 sample ELL students,
overall proficiency levels did not match their corresponding AZELLA scores. At
one of the sample districts, employees entered the overall proficiency levels
using a prior year’s rules, which had been changed. This resulted in 9 of the
26 sample students from that district being misclassified as ELL students.
􀁺􇩐 Program entry and exit—School officials must also record information about
the ELL students’ program entry and exit dates and reasons for exiting the
program. Accurate data on ELL students’ time in the program ensures that
they are only funded for 2 years, a requirement of HB 2064, which mandated
that the SEI models aim to achieve proficiency in 1 year. However, 6 of the
sample districts had questionable entry and exit dates, since 29 students
exited on the same day they enrolled or the next day. It is not clear how many
of these were students who simply pre-enrolled and then withdrew from the
school or program and how many were students whose data was entered
incorrectly by school staff.
School officials do not review data accuracy—Program officials at many
of the sample schools indicated that they did not confirm the accuracy of
assessment scores, and some were not familiar with the practice of confirming
program participation data within SAIS. ADE reminds district and charter school
employees at the end of the school year to check the data accuracy but does not
require districts and charter schools to confirm the data’s accuracy. According to
California state officials, California’s student data collection system requires
districts to confirm their data’s accuracy and perform procedures that identify and
resolve errors within their data.
Office of the Auditor General
page 25
ADE’s controls and checks for data reliability are
insufficient
When data submitted by districts and charter schools is inaccurate or incomplete,
ADE’s own internal processes should be able to identify the problems. Specifically,
ADE’s processes should not accept schools’ participation data for ELL students who
lack assessment data or who have assessment data indicating that they are already
English proficient. Auditors found that ADE’s integrity checks and controls for doing
so are insufficient. Consequently, the errors and limitations in the data are carried into
the system. These checks and controls are of two main types, and each needs
attention.
Weak integrity checks of ELL data may have resulted in improper
program funding—ADE is responsible for complying with statutory
requirements related to different types of student membership and program
funding. To help ensure appropriate funding levels, statutory requirements need to
be incorporated within SAIS and ensured through a data process called integrity
checks. As reported by the Auditor General in August 2006, ADE lacks a
comprehensive, documented process for ensuring that validation processes, such
as the ELL integrity checks, are functioning properly.1 Auditors identified several
failures in the integrity checks that may have resulted in improper program funding
and outcome reporting:
􀁺􇩐 Proficient students funded as program participants—Although A.R.S. §15-756
authorizes program funding only for students who test below the proficient
level, ADE’s fiscal year 2007 integrity checks do not prevent ineligible students
from being included in the calculations for ELL funding. Auditors identified 178
students, about 2 percent of the 10,054 sample ELL students, who had no
assessment scores in the current or prior year, or had an assessment score
indicating that, prior to entering the program, the ELL student was proficient.
Statutory requirements related to program funding should be translated into
SAIS integrity rules that function properly for all students. ADE needs to
change the system to check that the student received an assessment score
below the proficient level in the current or prior year before including the
student in funded participation counts.
􀁺􇩓 Students reclassified as proficient lack an assessment indicating
proficiency—Integrity checks do not ensure the accuracy of a key outcome
measure—the percentage of students becoming proficient. Auditors
determined that 27, or 4 percent, of the 662 sample ELL students who were
reclassified as proficient in fiscal year 2007 had assessment scores below
proficient or no assessment scores at all.
1 Office of the Auditor General’s performance audit report, ADE—Information Management, August 2006.
State of Arizona
page 26
􀁺􇩓 Significant variations in data on reclassification—Auditors also
identified significant variations in the percentages of sample
ELL students reclassified as proficient between fiscal years
2006 and 2007. Half of the sample districts and charter
schools had reclassification percentages that fluctuated more
than 20 percentage points between the 2 fiscal years. Such variation could
indicate data errors and should be further reviewed.
􀁺􇩏 Other integrity checks needed—Auditors identified two potential integrity
checks that, if implemented, might minimize observed errors. First, when
English is recorded as an ELL student’s home language, it could trigger a
warning that further investigation is required. Second, integrity checks should
ensure that proficiency levels correspond with the numeric assessment
scores.
Lack of process controls contributes to lost data—According to industry
best practices, an agency’s data system should include process controls to
ensure that no data is added, lost, or altered during processing. In August 2006,
the Auditor General’s performance audit of ADE’s information management
recommended that ADE add such controls. The presence of such controls could
help users and ADE prevent potential problems with SAIS data.
These controls could have assisted ADE in preventing and correcting a massive
state-wide recalculation and override of ELL counts that occurred in July 2007.
Because of to an internal processing oversight, over 20,000 ELL students were
excluded from the reported end-of-year funding counts—an $8 million error
eventually corrected by the override. Based on auditor analysis of ADE internal
communication, analysts struggled to identify the error’s cause. Further, although
the error could have affected prior year funding counts, ADE did not investigate
prior year discrepancies. ADE should have reviewed all funding data affected by
the processing error.
School officials also reported cases of ELL students who pass integrity checks in
the beginning of the year and later are excluded from the number of ELL students
for which funding was received. In the case of a charter school, after several
exchanges with charter school employees, ADE’s School Finance unit had to
reconcile the data manually. Other school officials also cited difficulties in
identifying data errors and their causes.
Determining the number of ELL students eligible for funding is a critical step in
processing ELL data and warrants additional process controls. SAIS could use
integrated system process controls to automatically check for discrepancies and
provide more information about how the ELL funding amounts were derived. Such
information would help districts and charter schools to review the accuracy of data
and would enable ADE and Auditor General staff to more effectively audit the data.
Office of the Auditor General
page 27
Two districts that reclassified over 30
percent of their ELL students as proficient
in 2006 reclassified 0 percent as proficient
ADE could expand types of data collected
Table 6 (on page 23) also identifies some of the data fields that ADE does not
currently collect, but for which information is available at the districts and charter
schools. Specifically,
􀁺􇩐 Program staffing data—ADE does not currently collect and maintain information
about the number and qualifications of ELL teachers. ADE’s Office of English
Language Acquisition Services requires districts to maintain ELL teacher
certification and endorsement documentation for review during monitoring visits.
This information could be systematically maintained state-wide for analysis and
monitoring purposes.
􀁺􇩐 Program ELD schedule data—ADE does not currently collect information about
the number of hours of ELD instruction provided to ELL students. Sample
districts and charter schools provided auditors with information on students’
hours of ELD instruction based on their proficiency and grade levels. Using the
program’s ELD schedules for ELL groups, hours of ELD instruction could be
calculated for individual ELL students and compared to their outcome
measures.
Recommendations
1. To ensure the accuracy of ELL data, ADE should work with districts and charter
schools to develop improved data submission and review processes.
2. To ensure proper ELL program funding, ADE should add ELL integrity checks
that require appropriate achievement data. ADE should not provide funding for
students without such data.
3. To improve data accuracy and auditability, ADE should implement process
controls that enable users, ADE, and the Auditor General to compare data totals
at critical points in the process, such as a list of students included in funded
participation counts.
4. To assist with analyzing ELL outcomes, ADE should consider collecting
additional data that describe how a program is implemented, such as teacher
qualifications and staffing levels.
State of Arizona
page 28
Office of the Auditor General
AGENCY RESPONSE
State of Arizona
Page 1 of 3
State of Arizona
Department of Education
Tom Horne
Superintendent of
Public Instruction
April 11, 2008
ADE General Comment
ADE welcomes recommendations that will lead to improved support for English
Language Learners, better accountability in program design and implementation, and
increased accuracy in data capture, management, and analysis.
Recognizing the limitations of the audit “Baseline Report of Arizona’s English Language
Learner Program” in so far that it is based on a random sample methodology and only for
the 2006-2007 Fiscal Year, ADE wishes to acknowledge the value of such an audit, and
in particular the general reference information that was provided. Such review of other
States’ and National data allows for a greater appreciation for the monumental steps
taken by the State of Arizona and the Department of Education in recent years in
addressing the needs of English Language Learners.
In additional, ADE wishes to further acknowledge that while the audit was limited to the
2006-2007 Fiscal Year, it was likewise limited in identifying major improvements and
achievements during the current fiscal year (2007-2008) for those areas cited. As such,
while ADE agrees to the four recommendations made by the Auditor General in this
report, ADE will also identify such improvements and achievements, as well as future
actions, in the response portions of this communication.
Page 2 of 3
ADE Comments and Responses Regarding Recommendations
Recommendation 1
To ensure the accuracy of ELL data, ADE should work with districts and charter schools
to develop improved data submission and review processes.
ADE Response
The recommendation of the Auditor General is agreed to and the audit recommendation
will be implemented.
ADE has an established training component that employs the STAR Team to provide
both general training as well as site specific assistance. ADE recognizes the difficulties
faced by school districts and charters regarding turn-over in personnel and will continue
to provide such assistance through the STAR Team in an on-going manner.
Since considerable data entry errors occur at the district and charter level, ADE is
currently working with the assessment/testing company to arrange all AZELLA scores to
be submitted directly to the SAIS System (electronic copies to the districts and charters)
thus eliminating a vast number of local entry errors.
ADE has formed a Data Subcommittee that meets every four to six weeks. The Data
Subcommittee is comprised of ADE staff (e.g. IT and R&E), and constituents from both
the K-12 and Higher Education communities. The Data Subcommittee will incorporate
dialog with a focus on accurate and timely data.
Recommendation 2
To ensure proper ELL program funding, ADE should add ELL integrity checks that
require appropriate achievement data prior to including the student in funded
participation counts.
ADE Response
The recommendation of the Auditor General is agreed to and the audit recommendation
will be implemented.
ADE will examine and evaluate the integrity checks to assure that each student eligible
for testing is tested and that English Language Learners classified as “proficient” are not
counted for funding.
Page 3 of 3
Recommendation 3
To improve data accuracy and auditability, ADE should implement process controls that
enable users, ADE, and the Auditor general to compare data totals at critical points in the
process, such as a list of students included in funded participation counts.
ADE Response
The recommendation of the Auditor General is agreed to and the audit recommendation
will be implemented.
To ensure the accuracy of the ELL management process, ADE will identify key
performance metrics resulting in total system auditability.
Recommendation 4
To assist with analyzing ELL outcomes, ADE should consider collecting additional data
that describe how a program is implemented, such as teacher qualifications and staffing
levels.
ADE Response
The recommendation of the Auditor general is agreed to and the audit recommendation
will be implemented.
For the past several months ADE has been working with software developers to design an
interactive software and technology piece for the field monitoring and data collection
process. The software and technology will allow field agents to collect such data as
matching model selection to actual model implementation, the number of hours provided
for by program design and actual program participation of individual students, teacher
qualifications, and other data traditionally collected in monitoring by paper and pencil.
The ability to transmit the collected data electronically to the Agency from the field is a
feature that is incorporated into the technology to better insure more timely and accurate
information regarding the ELL programs.
ADE will implement the software and technology in the monitoring and data collection
process.