Measures of student achievement and related outcomes: group 1 career ladder school districts

              I!
'I L
'I L
'I L
Le Measures of Student Achievement and Related Outcomes
'1 Group 1 Career Ladder School Districts
i
Edward F. Sloat
State Director of Research and Development
C. Diane Bishop
State Superintendent of Public Instruction
Arizona Department of Education
February 1994
Measures of Student Achievement and Related Outcomes
Group 1 Career Ladder School Districts
Edward F. Sloat
State Director of Research and Development
Edited by Joyce Hunter
Research and Development Division
February 1994
C. Diane Bishop
State Superintendent of Public Instruction
Arizona Department of Education
1535 West Jefferson
Phoenix, Arizona 85007
Table of Con tents
Overview of Findings ......................................................... ...................................................................1
Introduction ........ ... . .. . .. .. .. .. ..... .. .. . . .. ....... ..... .... ... ..... ....... . ... ... .. .. .. . . .. ... .. . .. .. . . . . . ... . . .. . .. . . . . . . ... . . . . .. . . . . . ......... 2
Methodology and Data Issues .............................................................................................................2.. .
Methodology. ..........................................................................................................................................2...
Issues of Experimental Design .......................................................................................... ......... .... ............3
Use of NCE Scores .................................................................................................................................4.. .
Use of Complete Composite and Subject-Specific Scores. .....................................................................4.. .
Calculation of Score Differentials ..................................................................................................... . .... ..... 4
Data Sources .............................................................................................................................................. 6
Student Performance Measures .............................................................................................................. 7
Historical Dropout and Graduation Rates ................................................................................................... 7
Student Achievement in CL and Non-CL Classrooms ............................................................................... 9
Aggregate Student Performance on the ITBS for CL and Non-CL Districts ........................................... 11
RePost Camr Ladder Student Performance Indicators .......................................................................... 13
Measures of Achievement on the Arizona Student Assessment Program (ASAP) Assessments ............. 15
199 1 Expected Versus Actual Test Scores for CL Districts. ........................................................ .......... . -16
Comparisons of 199 1 Expected Versus Actual Test Scores for CL and Non-CL Districts ........... ........... 17
Changing At-Risk Status ......................................................................................................................... 19
Appendix
Bibliography
Overview of Findings
1. Group 1 Career Ladder districts reported lower historical (1987 - 1992) dropout rates than Non-Career
Ladder districts.
2. Dropout rates for each racidethnic group were uniformly lower in Group 1 Career Ladder districts
than those found in Non-Career Ladder districts.
3. Graduation rates in Group 1 Career Ladder districts were 5 percent points above the state average in
1991 and 8 percent points above the state average in 1992. Comparatively, graduation rates for Non-
Career Ladder districts were 1 percent point below the state average in 199 1 and 2 percent points below
the state average in 1992.
4. Between 1988 and 1992 Complete Composite Nomal Curve Equivalent (NCE) scores on the ITBS for
each grade, 2 through 6, were between 5 and 10 percent above those reported by Non-Career Ladder
districts.
5. Group 1 Career Ladder NCE scores, aggregated across grades 2 through 6, were 7.95 percent higher in
1988,8.14 percent higher in 1990 and 9.10 percent higher in 1 99 1, than those in Non-Career Ladder
districts. Between 1988 and 199 1, the NCE score differential between Group 1 Career Ladder districts
and Non-Career Ladder districts increased.
6. For grade 3 Reading and Mathematics and Grade 6 Reading, the score differential between Group 1
Career Ladder districts and Non-Career Ladder districts increased between 1986 @re-Career Ladder
program implementation) and 199 1 (post-Career Ladder program implementation). However, this trend
was not apparent for grade 3 Language and grade 6 Language and Mathematics.
7. In 1990, NCE scores for students in grades K - 6 who received instruction from teachers participating
in the Camr Ladder program were 1.72 percent above scores reported for students not receiving
instruction h m Career Ladder teachers. NCE scores for students receiving instruction fiom Career
Ladder teachers were between 1.4 and 12.9 percent higher than those reported for students not
receiving instruction fiom Career Ladder teachers in 10 out of 13 (77 percent) Grot*;, 1 Career Ladder
districts. (Note: no data was available at the classroom level for one of the Group 1 districts.)
8. For the Group 1 Career Ladder districts, student performance in each subject area of the March 1993
ASAP Form Dl assessments for grades 3 and 8 was between 1.4 and 6.8 percent above Non-Career
Ladder districts.
9. Student achievement (1991 ITBS Complete Composite NCE scores for grades K - 6) in Group 1 Career
Ladder districts exceeded expected performance when adjusted for community wealth and
studentldistrict population characteristics. Expected performance in Non-Career Ladder districts was
not significantly different fhm actual performance. (Note: evaluation of expected versus actual
performance was significant at the 95 percent confidence level.)
10. Group 1 Career Ladder districts which were identified as not being at-risk performed at higher than
expected levels, while Group 1 Career Ladder districts identified as being at-risk performed slightly
below expectations.
11. Group 1 Career Ladder districts which were identified as not being at-risk in 1990 grew to be relatively
more at-risk by 1992. Group 1 Career Ladder districts which were considered to be at-risk in 1990 did
not become significantly more at-risk by 1992.
Introduction
At the request of the Arizona Department of Education (ADE) Career Ladder Office, staff of the ADE Research and
Development Division were asked to investigate student outcome measures for Arizona school districts participating
in the state's Career Ladder (CL) program. The investigation focused on identifying variations in selected outcome
measures between participating and non-participating districts.
This report summarizes the empirical information and analytical findings of the project. Part 1, Methodology and
Data Issues, discusses methodological assumptions and decisions made at the start of the project, including issues
concerning achievement scores, presentation of comparative data and primary sources of information. Part 2, Student
Performance Measures, presents tabulations and charts of comparative outcome measures organized by type of
analysis.
Part 1
Methodology and Data Issues
Methodology. At the beginning of the project, staff fiom the ADE Research and Development Division and the
Camr Ladder Office met to identify the type and scope of the analysis and the availability of relevant data sets. The
following parameters were defined to guide the study:
First, it was initially agreed that an historical perspective would be important, tracking various outcome
measures over time in an effort to delineate trend differences between CL and non-CL districts.
Second, it was felt that while multiple outcome measures should be investigated, the emphasis would be
placed on student achievement.
Third, discussions among ADE staff and representatives fhm CL districts suggested that outcome measures
should be looked at in a variety of ways, including within and between comparative CL districts as well as
contrasted against non-CL districts.
Fourth, because the primary concentration of teacher participation is at the elementary grade levels, the
analysis of student achievement is restricted to grades K - 6. The exception to this would be Arizona Student
Assessment Program (ASAP) assessment results which include grades 3 and 8.
Fifth, it was decided not to focus on individual district outcomes. Therefore, most of the data presented in
this report compare averages aggregated across all CL and non-CL districts.
Finally, since participation in the CL program began in 1987, it was decided to limit the analysis to districts
which have been in the program the longest. Thus, the study group was restricted to the following original
14 Group 1 CL districts.
Group 1 Career Ladder Districts
Amphitheater
Apache Junction
Catalina Foothills
Cave Creek
Creighton
m a r t
Flowing Wells
Ganado
Kyrene
Litchfield
Mesa
Peoria
Sunnyside
Window Rock
Due to limitations in staff resources and available data, the type of student outcome measures investigated were
restricted to readily available indicators, including student performance on ITBSrrAP tests, the Arizona Student
Assessment Program (ASAP) assessment scores and high school (grades 9- 12) dropout and graduation rates.l
Analysis of student outcomes incorporated the following investigations using CL and non-CL district data:
Aggregate student ITBS achievement data by grade,
Student achievement on the ITBS by classroom,
1993 ASAP Form Dl student assessment results by subject and grade level,
Mpost ITBS achievement by grade level,
Expected versus actual ITBS student achievement, and
Trends in dropout and graduation rates.
District-level analyses of both ITBS and ASAP test scores were based on weighted average aggregations of student-level
data2 A final decision was made to restrict the analysis of ITBS results to the elementary grades in order to
construct comparable district- and classroom-level aggregation^.^
Issues of Experimental Design. The focus of this report is necessarily limited to the observation of various student
outcome indicators associated with Group 1 CL and non-CL diicts. Since the Group 1 CL districts were
determined prior to the start of this evaluation, it was not possible to utilize strict experimental methodologies which
would allow for a complete understanding of the causal relationship between implementation of the CL program and
varialions in student performance measures.
Participation in the CL program by Arizona school districts, school sites and classroom teachers was not a random
event. Application and awards of limited program funding restricted participation to those districts, schools and
teachers who requested, and were allowed, to participate. Throughout the history of the program, additional districts
and school sites have been incorporated. Thus, statistically motivated inferences based on pre-selected experimental
and control group evaluations are not possible. For this analysis, too little is known about the characteristics of the
Group 1 CL districts as they relate to non-CL districts to assume that differences in empirical measurements are due
solely to the effects of the CL program. Factors such as the lack of available data on competing at-risk programs,
student involvement in multiple education support programs, teachertschooLldistrict characteristics which initiated
application to the CL program, and student outcome measures attributable to each of these factors, make parsing of
performance and outcome indicators to the CL or any other education program problematic.
The difficulty of attributing quantifiable variations in student outcome measures to broad-based education programs
is not unique to the CL program. Evaluations of programs such as Chapter 1, Chapter 2, School Lunch, At-Risk,
Dropout Prevention, etc., are all characterized by an inability to strictly measure their impacts on student
achievement. Indeed, most public policy and programmatic evaluations are conducted with the understanding that the
complexity of competing factors (human, policy or program) precludes any evaluation of effectiveness based purely
on empirical findings. Rather, observations of measurable events are made to supply the evaluator with additional
information which facilitates an understanding of a particular belief or hypothesis. When such empirical data are
combined with other qualitative information, observations and understandings, it becomes possible to reach more
informed conclusions about the causal impact of a particular program or activity.
1 (ITBSITAP) Iowa Tests of Basic Skills and Tests of Achievement and Proficiency, Riverside Publishing
Company, Chicago, Illinois.
2 Calculations of weighted average achievement levels are used throughout this report. The number of students
within an analysis group (such as a classroom, district or group of districts) serves as the weight. This weight (or
student count) is applied to the average test score of the analysis group in order to reflect its relative importance
compared to other group averages.
3 Students in grades 7 - 12 traditionally do not reside in self-contained classrooms, thus preventing a one-to-one
link between tests scores and the teacher identifier available from the computer record.
3
This report is intended to provide a variety of empirical observations on the variation and relationship of student
outcome indicators for Group 1 CL and non-CL districts. It is by no means comprehensive. Much of the analysis has
been restricted due to the lack of available data and resources. Hopefully, the comparative evaluations presented will
lead to additional questions and areas of interest and, ultimately, to the implementation of a more exacting
experimental design.
Use of NCE Scores. For the purpose of this report, all ITBS student performance figures are based on normal c w e
equivalent (NCE) scores. NCE scores were utilized for their mathematical and computational properties which allow
for within- and across-grade comparisons of student performance. Unlike percentile ranks and grade-equivalents,
NCE scores may be averaged across subpopulations of students within specific grades, schools or districts or across
multiple grade levels.
An NCE score indicates the relative performance of an individual compared to the distribution of test scores
achieved by a national sample of students - commonly called a norm group. NCE scores range between 0 and 100
points. For example, if an Arizona student in a specific grade achieves a score which is equal to the average score of
students at the same grade level represented in the national norm group, the Arizona student would receive an NCE
score of 50. If the Arizona student achieved an NCE score greater than the average score of the norm group, the
Arizona student's NCE score would be above 50. Finally, if the Arizona student earned a score lower than the
average of the norm group, the resulting NCE score would be less than 50. This interpretive quality of NCE scores
holds for an individual student or for groupings of students, schools or districts.
Use of Complete Composite and Subject-Specific Test Scorn. The focus of this study was to investigate the
relative performance levels of students in CL and non-CL di~trictsN.~o a priori hypothesis was stated on the effect of
the CL program on student performance within a particular subject area Thus, emphasis was placed on analyzing the
composite scores of students because they represent an overall relative performance level incorporating the subject
areas of Reading, Mathematics, Language and a variety of smaller subdomains of knowledge and skills. In fact, it is
assumed that if the CL program is effective, the effect will be shown across all subject areas.
In one instance, the use of Complete Composite scores was not possible. In compiling test-score information for the
1986 school year, it was found that computer-based files were no longer available. As a result, data for that year had
to be extracted fiom published reports. Unfortunately, the Riverside Publishing Company, which publishes the
Arizona ITBSrrAP test scores, did not provide accessible hard-copy reports of Complete Composite scores on a
district-by-grade level for the 1986 school year.5 The only information readily available was for the primary subject
areas of Reading, Language and Mathematics. As a result, the section comparing student performance in 1986 to
1992 reports weighted average NCE scores by subject area
Calculation of Score Differentials. Throughout this report, two methods of presenting weighted average NCE
scores are used to compare CL and non-CL districts. In many instances, the actual NCE scores of these groups are
reported. This tells the reader about each group's overall average achievement level. In addition, comparing average
NCE scores gives some indication of the relative position of one group to the other. However, it does not clearly
demonstrate the degree to which the scores differ. This becomes even more problematic when comparing actual NCE
scores over time.
To view the relative performance levels of CL and non-CL districts, calculations of score differentials are used.
Score differentials simply report the percentage difference in absolute NCE scores between the two groups. In all
cases, the weighted average scores for CL districts are compared against non-CL districts. Thus, positive differentials
indicate that CL average scores exceed those of non-CL districts, while negative differentials indicate that CL
d i c t s performed at a level below non-CL districts. For example, a score differential of +3.20 percent indicates that
the actual NCE score for CL districts was 3.20 percent above that for non-CL districts. Similarly, a differential of
-3.20 percent indicates that the score for CL districts was 3.20 percent below that for non-CL districts.
4 In essence, the empirical hypothesis being testing is that there is no difference in overall student performance
between the two groups. Conventions and properties of classical statistical inference hold that a test of
hypothesis be stated in terms of a "no difference" condition. This is commonly denoted as the "null" hypothesis.
5 Individual district reports retained in archives which contained additional subtest information were not retrieved
due to an inability to manually compile and compute the Complete Composite scores across grades and subjects.
4
In the following illustration, Figure 2 reports the actual weighted average NCE scores of two hypothetical groups in
1987 and 1992.
Figure 2
GROUP GROUP GROW GROW
1 2 I 2
As shown above, the scores for Group 1 exceeded those of Group 2 in both years. I . addition, the scores for both
groups declined between the two time periods. However, h m the information presented in Figure 2, it is difficult to
ascertain both the relative achievement levels of the two groups for each year and how these achievement levels have
changed ova time.
In contrast, Figure 3 expresses the NCE scores for both groups as differentials. The positive differentials are
interpreted as follows: (1) Group 1 scores exceeded Group 2 in both reference years; (2) in 1987, Group 1 scores
were 10.29 percent higher than those for Group 2; (3) in 1992, Group 1 scores were 1 1.17 percent higher than Group
2 scores; and (4) the difference in Group 1 scores increased by .88 percent points between 1987 and 1992. Thus,
while the level of achievement for Group 1 exceeded that for Group 2 in both years, the degree of this difference also
increased.
NCE SCORE DIFflERENTIALS
Data h u m s . To investigate ITBS achievement, historical information sets were constructed from computerized
data files maintained by the ADE for the 1988, 1990 and 1991 school years. Changes in the ITBS testing program
prevented the use of 1992 and 1993 data .6 Each yearly data file contained student-specific information, including
teacher's name, grade level, school designation, demographic data and standardized test scores.
Selection criteria were developed to extract ITBS 'scores only for grades K - 6, because it was not possible to link
teacher identifiers with student test results for grades 7 - 12. District identifiers were then constructed which
identified CL and non-CL districts for use in aggregate calculations. At the classroom level, teacher lists were
obtained for each CL district. From these lists, the names of teachers participating in the CL program were matched
to teacher names maintained on the individual student records.
To compare student achievement levels observed under the Career Ladder program with levels existing prior to its
inception in 1987, data firom the 1986 school year was used. Since no computer data files existed for this school year,
staff extracted ITBS grade-equivalent scores for grades 3 and 6 by subject area Erom the Appendh. to the Statewide
Reportf or Arizona Pupil Achievement Testing, June 1986.R~e source limitations and computational difficulties
prevented extraction of additional grade and subject information.
AII of the ITBS analyses utilized 1988-normed NCE scores.8 As mentioned above, it was preferable to use NCE
scores due to their mathematical properties and ease of interpretation. All of the 1988 - 91 data were based on 1988
norms. However, the 1986 data were available originally only for 1982 norms. Thus, it was necessary to convert
these data to the equivalent 1988 benchmark year. This was accomplished as follows:
1. Tabulate 1986 grade-equivalent scores by subject for grades 3 and 6 for each Group 1 CL district, and
2. Use conversion tables provided by the Riverside Publishing Company to map 1986 grade-equivalent
scores to 1986 percentile ranks, map 1986 percentile ranks to 1988-norm percentile ranks, and map
1988-norm percentile ranks to NCE scores.
Student performance data on the Arizona Student Assessment Program (ASAP) March 1993 assessments were
compiled for grades 3 and 8 h m information sets maintained by the ADE Research and Development Division.
Grade 12 assessment scores were not available for two of the Group 1 CL districts and three more districts did not
have active 9-12 grade levels. In addition, high school teacher participation rates in the CL program for some of the
districts were substantially lower than those found in the elementary grades. Finally, as with ITBS results, it is not
possible to obtain comparative teacher participation and student assessment scores for grade 12. For these reasons,
grade 12 ASAP assessment results were not included in the analysis.
In addition to looking at trends in actual student test scores, a predictive model of student performance was utilized.
This model was developed by the ADE Research and Development Division to estimate the effects of numerous
community, economic and population factors on variations in district-level ITBS test scores. The model incorporated
information obtained firom the U.S. Bureau of the Census and the ADE's report on the At-Risk Status of Arizona
School ~istricts?
ti In 1992, Arizona law altered the structure of the Arizona Pupil Achievement Testing Program by moving the
administration of standardized testing from the spring to the fall, reducing the number of grade levels tested and
restricting the subject areas to selected subtests within Reading, Language and Mathematics. Due to these
changes, no comparable Complete Composite ITBS scores for grades K - 6 could be constructed.
Appendir to the Statewide Reportfor Arizona Pupil Achievement Testing, Arizona Department of Education,
Phoenix, Arizona, June 1986.
8 Refer to Part 1, Use of NCE Scores.
9 The Arizona Depamnent of Education participated in a joint project with the U.S. Bureau of the Census and the
U.S. Department of Education that electronically mapped school district boundaries throughout the state. The
electronic boundary files were then merged with 1990 Census information to provide a wide variety of
demographic and economic tabulations by school district. The Census data used in this report were extracted
from the School District Data Book CD-ROM, U.S. Bureau of the Census, October 1993. Information on the at-risk
status of school districts was obtained from USTAT: The At-Risk Status ofArizona's School Districts,
Arizona Department of Education Research and Development Division, March 1992.
6
The predictor variables used in estimating aggregate test scores included the following:
Median Family Income
Median Value of Owner-Occupied Housing Units
Percent of Students Eligible for Participation in the Freel'educed Price Lunch Program
Percent of Students Identified as Being Limited English Proficient
District Absentee Rate
District Index of Mobility
Percent of Minority Students in the District
Percent of Students Who Have a Computer at Home
Part 2
Student Performance Measures
Historical Dropout and Graduation Rates. Figure 4 displays weighted average annual high school (grades 9 - 12)
dropout rates for CL and non-CL districts for 1986 through 1992.1° The figures reported are calculated annually,
based on a nine-month, fall-to-spring school year.
Figure 4
NINE-MONTH HIGH SCHOOL (S-12) DRO#WTT RATE
CA- UDDm Vb N O W = LADOW DISTRICTS
1986 THROUGH 1992
As shown above, with the exception of the 1986 school year, CL districts reported lower average nine-month high
school dropout rates than non-CL districts. Beginning in 1987, CL district dropout rates steadily declined from a high
of 9.65 percent to a low of 7.56 in 1992. The largest difference in dropout rates occurred in 1989 when the rate for
CL districts was 1.86 percent points below that of non-CL districts. The detailed dropout figures are provided in
Figure 5 below.
-
10 Historical dropout rate information was provided by the ADE School Finance Unit. Detailed data for 1992 may
be found in Caryn R. Shoemaker's Dropout Rate Study, 1991-92, A Study ofAnnua1 Dropout Rates in Arizona
Public Schools, G r a h Seven Through Twelve, Arizona Department of Education School Finance Unit,
Phoenix, Arizona, February 1993.
7
Figure 5
Nine-Month High School Dropout Rate
(Percent)
Group 1 Career
11.75
9.65
9.32
8.69
8.76
8.55
7.56
Non-Career Ladder
Districts
10.86
10.86
1029
10.55
10.20
8.95
9.17
Figure 6 reports 1992 nine-month high school dropout rates by racelethnicity for CL and non-CL districts. As shown,
both CL and non-CL districts displayed the same pattern of midethnic dropout rates. For both groups, White and
Asian student populations reported lower annual proportions of dropouts than Black, Hispanic or Native American
populations. However, the data also indicated that CL districts displayed lower dropout rates within each
racidethnic category than non-CL districts.
Figure 6
mite Blrk m c AUAN ArnnlPI
Figure 7 presents weighted average 199 1 and 1992 graduation rates aggregated for CL and non-CL districts. l1 As
indicated, high school graduation rates for CL districts exceeded those of non-CL districts in both years. In addition,
CL district graduation rates exceeded the state average rate by 5 percent points in 1991 and by 8 percent points in
1992. Comparatively, non-CL districts declined from a 1 percent point advantage over the state average in 1991 to 2
percent points below the state average in 1992.
Figure 7
Weighted Average Graduation IZntesl2
-1991 -
Difference
Percent JmILsMe
Group 1 Career
Ladder Districts 70% +5% Pts.
Non-Career
Ladder Districts fwl -1% Pt.
State Total 65%
- 1992 -
Difference
Percent l?JmnB&
Wl -2% Pts.
Student Achievement in CL and Non-CL Classrooms. As part of the investigation into the impact of the CL
program on student achievement, ADE R&D staff mapped within-district teacher participation to individual student
performance records. This analysis was performed using 1990 ITBS computer data files for each of the 14 Group 1
CL districts. The data represents Complete Composite NCE scores aggregated across grades K - 6.
Figure 8, on the following page, reports the within-district difference betwttn classrooms with and without
participating CL teachers. As shown, 11 of 13 Group 1 CL districts reported that the classrooms with participating
CL teachers have higher student achievement levels than those with non-participating teachers.13
1 Information on district graduation rates was provided by the ADE School Finance Unit. Detailed district data are
available in Caryn R. Shoemaker's Class of 1992 Graduation Rate Study. A Study ofGraduation Rates for
Arizona Public High Schools, Arizona Department of Education School Finance Unit, Phoenix, Arizona, March
1993.
l2 Changes in data definitions over the two study years suggest comparison of the graduation rate figures over time
should be done with caution.
13 No classroom data were available for one CL district. No adjustments were made for the level on which CL
teachers participated in the program. Generally, teachers in lower levels of the program are less experienced. It
is possible that student performance in CL districts with larger proportions of lower level CL teachers may not
be impacted to the same degree as it is in districts with more higher level CL teachers.
9
Figure 8
PWCW D I M C E C A M LADDER AND NON-CAM LADDER
CLAssmoMS
rsmo rres c o m m COMPOSITE NCE SCMIES
GRADES K - 6 BY DlSTRlCT
In 1990, Group 1 CL districts were composed of approximately 86,369 K - 6 students, 29 percent (24,699) of whom
were contained in classrooms with participating teachers (see Figure 9). The weighted average NCE score of students
within CL classrooms was 51.775, while non-CL classroom students reported a weighted average score of 50.898.
Thus, across all Group 1 CL districts, the NCE score for students receiving instruction fiom CL teachers was 1.7
percent higher than the NCE score for students in non-CL classrooms.
Figure 9
1990 ITBS Scores for Students in CL and Non-CL Classrooms
Group 1 Carnr Ladder Districts
Number of Weighted Average
Students Percent N C E 1 4
K-6 Students Receiving Instruction from Teachers
Participating in Group 1 CL Program: 24,699 28.6% 52.470
K-6 Students Receiving Instruction from Teachers
Not Participating in Group 1 CL Program: 61.662 7r.4./0 51.224
Total: 86361 100.0%
l4 Figures reflect 1990 ITBS Complete Composite NCE scores.
Aggregate Student Performance on the ITBS for Career Ladder and Non-Career Ladder Districts. Figure 10
reports the weighted average NCE score for Group 1 Career Ladder and Non-Career Ladder districts for grades 2
through 6. Grade 1 scores were not reported due to changes in the ITBS program during the 1990 and 1991 school
years which made aggregations at the substate level inappropriate.15 All scores reflect ITBS Complete Composite
achievement levels which combine subject area scores for Reading, Language and Mathematics.
Figure 10
Weighted Average NCE Scores For Career Ladder and Non-Career Ladder Districts
Wed on Grades 2 - 6 ITBS Complete Composite Scores, Using 1988 Norms
Career
Ladder
Districts
Non-Career
Ladder
Districts
Percent Difference
in .
School Gmk IYcEhSa
Composite
Composite
Composite
Figure 10 above shows that for all grades in each reference year, CL Met scores exceeded those of non-CL
districts. The "Percent Difference in NCE Scores" column reports the score differentials which were calculated to
reflect the relative achievement levels of students within CL and non-CL districts. The positive values indicate that
the average CL district scores are above the average non-CL district scores in all cases. Conversely, negative values
would have indicated that CL districts performed at a level below non-CL districts. A grade-by-grade representation
of this data is reported in Figure 1 1 for the 1988, 1990 and 199 1 school years.
'5 In 1990, student achievement scores for grades 1 and 12 were estimated using a statewide sampling of students
and schools; therefore, inferences are not valid at the substate level or when data are aggregated into district
clusters.
11
Figure 11
As shown above, the relative difference in NCE scores between CL and non-CL districts grew between 1988 and
1991 for each grade except grade 4. The largest score diffmntial, a 10.25 percent difference for CL over non-CL
scores, occurred in grade 2 during 1991. Note that both the 1990 and 1991 score differentials tend to decline as the
grade level increases.
By using NCE scores, it is possible to combine grade-specific achievement data into a single aggregate
performance level representative of all students in grades 2 - 6 within a district. Figure 12 reports the aggregate
weighted average NCE scores for CL and non-CL districts by school year.
Aggregate Weighted Average NCE Scores
CL and Non-CL Districts
Grades 2 - 6, ITBS Complete Composite
Career NOD-Career
Ladder Ladder Percent
Districts Districts
Figure 13 on the next page reports the above information in terms of score differentials; it shows, on average, that:
1. Student ITBS performance in the elementary grades for CL districts exceeded that of non-CL districts
for each referenced year, and
2. The degree to which CL districts outperformed non-CL districts has iricreased over time.
12
Figure 13
PMeent Diffemnce in Complete Compodte NCE Sconw
Clmr Ladder vs. NorrCPrear Ladder Districts
1988,1990 and 1991 ITBS, Glpder 2 - 6
In summary, weighted average NCE scores in CL districts exceeded those of non-CL districts for grades 2 through 6.
Further analysis shows that the differential in these scores has been increasing over time. Finally, aggregate student
scores in the elementary grades in CL districts also have exceeded those of non-CL districts, with the differential
growing fiom 7.95 percent in 1988 to 9.10 percent in 1991, as shown in Figure 13 above.
PrJPost Career Ladder Student Performance Indicators. Program participation of the Group 1 Career Ladder
districts began in 1987. In order to evaluate student performance in a prelpost CL participation framework, ADE
F@D staff compiled ITBS Complete Composite test score information for grades 3 and 6 by primary subject areas
(Reading, Language and Mathematics) for the 1986 and 1991 school years. Figure 14, on the following page, reports
the comparative NCE scores for these two years.
Figure 14
Grade 3
Reading
L~~guage
Mathematics
Grade 6
Reading
L=%uage
Mathematics
Student Performance by Subject for 1986 and 1991
ITBS Complete Composite Weighted Average NCE Scores
198-
CL Non-CL Percent
DistrietsDistricts-
1991
CL Non-CL Percent
Districts Districts
As shown above, for each subject area CL district scores were higher than non-CL district scores in both 1986 and
1991. Figure 15 below displays the score differentials between the CL and non-CL districts. As stated before, the
positive differentials indicate that CL district scores exceeded those of non-CL districts; negative differentials would
have indicated the reverse.
Fire 15
PWCEFIT Dl-CE IN WEGHTB) AVeUGE NCE SC-
1986 AND 1991
CA- IADDW VS. NON-CA- IADDW DISTRCTS
ITBS COMPLETE COMPOSITE BY GRADE AND SUBJECT
In three of the six subject areas (grade 3 Reading and Math, and Gradt 6 Reading) the degree to which CL district
test scores exceeded non-CL district scores increased between 1986 and 199 1. Relative performance levels for Group
1 CL districts appear to be higher for grade 3 than for grade 6. That is, score differentials for the CL districts were
higher in 1991 in two of the three subject areas for grade 3. Comparatively, this trend was observed in only one of
the three subject areas for grade 6.
Lower relative performance levels for the grade 6 subject areas compared to the grade 3 subject areas within Group 1
CL districts may be due to a difference in the instructional environments. In all districts, grade 3 students remain in a
single classroom. However, in grade 6, some elementary schools allow students to move between classrooms
throughout the day. This may mean that these students are instructed by several teachers, some of whom are not CL
teachers. If this situation occurs more fkquently in Group 1 CL districts, it may tend to mitigate the effect of CL-related
instruction on the overall grade 6 student population. However, no explicit information on this situation or its
potential effect is available.
Another possible explanation for lower relative performance in grade 6 subject areas may be that the impact of CL-related
instruction declines as the age and grade of the students increase. That is, student performance may be less
directly affected and influenced by CL-related strategies at higher agelgrade levels in comparison to peer-, home-,
community- and school-based factors.
Measures of Achievement on the Arizona Student Assessment Program (ASAP) Assessments. Using an
analysis similar to the one used to investigate performance differences on the ITBS, R&D staff compiled student
achievement scores on the 1993 ASAP Form Dl assessments for both CL and non-CL districts. Figure 16 reports the
average student scores for CL and non-CL districts for each ASAP subject area. As shown, CL districts performed
better than non-CL districts in both grades 3 and 8.
Figure 16
1993 ASAP Form Dl Assessments
Grades 3 and 8 Average Score by Subject Area
CL and Non-CL Districts
Reading Mathematics Writing
Gl3uM
Camr Ladder 9.3 1 11.75
Non-Career Ladder 8.89 1 1.59
Percent Difference 4.72% 1.38%
Reading Mathematics Writing
G m u f20 -. ~~ L16 -- P Q S i w
Career Ladder 1 1.20 5.3 1
Non-Career Ladder 10.70 4.97
Percent Difference 4.67% 6.84%
Figure 17 below reports the difference in mean ASAP Form Dl assessment scores by subject area for CL and non-
CL districts. Positive values indicate higher mean scores for CL districts, while negative values indicate poorer
performance compared to non-CL districts.
F i r e 17
Percent Difference in March 1993 ASAP Mean Assessment Scores
Career Ladder Versus NOD-Career Ladder Districts
Writing
As shown above, CL districts outpe!rformed non-CL districts in grades 3 and 8 for all subject areas. Additional
analysis on the distribution of scores indicated that CL districts reported higher proportions of students scoring in the
upper 50 percent of possible assessment points.
1991 Expected Versus Actual Test Scores for CL Districts. As part of the analysis, M D staff utilized a statistical
modeling h e w o r k which incorporated empirical information used in measuring at-risk student populations, non-test
indicator information fiom the Arizona Student Assessment Program and data recently released from the U.S.
Bureau of the Census. In this framework, variations in district ITBS test scores were associated with a variety of
community wealth and district economic indicators such as Median Family Income, Median Value of Owner-
Occupied Housing Units and Percent of Students Who Have a Computer in the Home as well as selected district and
student population characteristics, including the Percent of Students Eligible to Participate in the FredReduced
Price Lunch Program, Percent of Students Identified as Being Limited English Proficient, Percent of Minority
Students, the district Absentee Rate, and district I& of Mobility. l6
Using this information, predicted ITBS scores were generated and compartd with actual observed student
performance measures. A selected summary of this information generated for the Group 1 CL districts is provided in
Figure 18, which follows. l7
l6 Data sources included preliminary information from the School District Data Book, U.S. Bureau of the Census
School District Census Mapping Project; Ed-STAT: The At-Risk Stam of Arizona's School Districts, Arizona
Department of Education Research and Development Division, March 1992; and March 1993 Assessment
Results, State ofArizona, Arizona Student Assessment Program, Arizona Department of Education, September
28, 1993.
l7 The table presented makes a distinction between at-risk and not-at-risk school districts. An at-risk district has a
composite risk index greater than or equal to 0.00. A not-at-risk district has a composite risk index of less than
0.00.
Figure 18
P U C m W Y y * . Y Yo.ummmlAm
~ICUUlIAD.P.LINCIS
. * p . o N m 1 ~ 1 I ( C L ~ C W I 1 K - L
(ALL naRLI -EL. AS A-The
table above presents an ordered listing of districts according to their relative at-risk index (as indicated by the
figures at far right). Districts with at-risk indexes of less than 0.00 are identified as being relatively not-at-risk. The
opposite is true for districts with indexes greater than zero. The mathematical properties of the at-risk calculations
position the state average at zero.
Among the individual Group 1 CL districts, 8 of 14 (57 percent) report actual performance levels exceeding
predicted scores. That is, the predicted score is less than the actual score. This indicates that pupils in these districts
performed better than expected given the characteristics of the community and student population. The CL districts
that did not perform above expectations are distributed across the range of at-risk values.
The table above also segments the model information into two discrete categories: one for relatively not-at-risk
districts (indexes less than 0.00), the other for comparatively at-risk districts (indexes greater than 0.00). Review of
the model estimates suggests that CL districts which are relatively not-at-risk tend to perform at higher than expected
performance levels. The reverse is true for comparatively at-risk districts.
Comparisons of 1991 Expected Venus Actual Test Scores for CL and Non-CL Districts. Results fkom the
modeling process were also generated for non-CL districts. Comparisons of actual and predicted test scores as well
as selected model variables are presented in Figure 19, which reveals that students in CL districts performed better
on the ITBS than students in non-CL districts, receiving higher actual student achievement scores than those
predicted by the model. In this case, actual 199 1 ITBS performance for CL districts on the Complete Composite for
grades K - 6 was 1.45 NCE points above model expectations. This is compared to .417 points for non-CL districts
and .479 points for all districts combined. l8
18 Predicted NCE scores for non-CL districts are not significantly different at the 95 percent confidence level
@ = .05) fiom actual scores. Differences in predicted and actual scores for CL districts are significant at the 95
percent level.
Figure 19
Scfccted Summary Indiuton for Camr Ladder vr Non-Camr Ladder Districts l9
(1990 lTBS Model Eatimata ~ n f o r m a t i o n ~ ~
1991 Expected Venus Actual T a t Scorn
Predicted Actual Diflerenee
d l Iyslsu m
No& Districts Man 45.824 4624 1 .417
~ o u n p 188,653 192,357
Std. Dcv. 6.%2 7.530
CL Districts Man 49.727 51.177 1.450
Count 86,357 86,357
Std. Dcv. 5.789 6.898
All Districts Man 46.728 47208 .479
Count 254,313 258,017
Std. Dcv. 7.035 7.728
Scicctcd At-Risk Indiuton
Minority
At-Risk FIR ~ n n c h ~ ~ LEP Students
IPPu IPPu lPPu - No.CL Diitricts Mmn -0.64 -0.35 -0.12 41%
Cmnt 196,950 189,078 192,184 195357
SM. Dcv. 2.61 0.88 0.71 27
CL Districts Man -2.13 -0.86 -0.36 27%
Count 86,357 86,357 86,357 86,357
Std. Dcv. 2.5 1 0.71 0.32 2 1
Selected Walth and Income Indicators
Median Mediin M&O
Family Housing Per Pupil
Income Valua Expenditures
W W W
No& Districts Man S30,%7 572,678 S2.893.86
Count 192,357 191.749 1%,950
Std. Dcv. $8,900 S23,568 S2M.83
CL Districts Man S35,260 S84.717 $2,918.33
Count 86,357 86,357 86,357
Std. Dcv. $8,575 522,020 S228.12
19 All values represent weighted averages based on K - 6 student counts.
20 Observations of Census estimates on family income and on housing values were restricted to districts having
household sample sizes greater than 60 and a household sampling rate of greater than or equal to 10 percent.
21 199 1 ITBS Complete Composite NCE Scores for grades K - 6.
22 Lack of complete information excluded some district records from the regression model, resulting in lower
student counts than those reported for the actual NCE score.
23 F/R Lunch: an index for percentage of students eligible for federal Free and Reduced Price Lunch program.
18
It should be noted that the predicted score for non-CL districts is not statistically different from the actual score.
However, the predicted and actual scores for the CL districts were significantly different from each other, supporting
the conclusion that students in CL districts performed at higher than expected levels.
Reviewing some of the selected descriptive data used in the modeling process revealed that, as a group, CL districts:
1. May be considered to be relatively less at-risk than non-CL districts, reporting an average at-risk index of
-2.13, compared to -.64 for the non-CL districts;
2. Contain lower proportions of students eligible for the Free or Reduced Price Lunch program than non-CL
districts;
3. Contain lower proportions of limited English proficient (LEP) students;
4. May be characterized as having higher median family incomes and higher median housing values;
5. Contain lower proportions of minority students; and
6. Report about the same level of expenditures per pupil for maintenance and operations.
Changing At-Risk Status. In 1987, the ADE Research and Development Division constructed an empirical
measure of relative at-risk conditions of student populations in Arizona school districts.24 This measure is in the form
of an index in which 0.00 indicates the state average, positive values reflect relatively higher (more) at-risk
conditions and negative values indicate relatively lower (less) at-risk conditions. For this report, districts with at-risk
indexes below 0.00 are considered not-at-risk while districts with indexes above 0.00 are considered at-risk. .
Within a given year, comparisons of at-risk indexes provide a general understanding of the characteristics within
diffmnt district student populations. Examined over time, these indexes reveal how these characteridcs have
changed. Figure 20 examines the change in the relative at-risk rauking for each of the Group 1 CL districts between
1990 and 1992.
Figure 20
At-Risk Status of Group 1 CL Districts in 1990 and 1992
1990
At-Risk
Index
1990
At-Risk
Ranking
1992 Change in
At-Risk At-Risk
Ranking Ranking
DISTRICT
CATALINA FOOTHILLS
KYRENE
LITCHFIELD
PEORIA
CAVE CREEK
MESA
AMPHITHEATER
FLOWING WELLS
APACHE JUNCTION
07-02-89 DYSART .8875
1 0-02- 12 SUNNY SIDE 1.459 1
01-02-08 WINDOW ROCK 2.8142
07-04- 14 CREIGHTON 3.0369
0 1-02-20 GANADO UNIFIED 3.0435
24 EdSTAT: The At-Risk Status of Arizona School Districts, Arizona Department of Education Research and
Development Division, August 1991. Rankings based on 1987 data were not used in this analysis due to
definitional changes in the at-risk variables.
During this time, 71 percent (10 of 14) of the Group 1 CL districts experienced an increase in their at-risk ranking,
indicating that these districts became relatively more at-risk in 1992 than they were in 1990. Eight of these districts
wm initially considered to be relatively not-at-risk by virtue of their negative 1990 index. Only two of the 10
Group 1 CL districts which became relatively more at-risk began with an at-risk index in the positive range. Looked
at in a slightly different way, there were nine CL districts that were considered to be not-at-risk in 1990. Eight of
these districts grew more at-risk by 1992. In contrast, there were five CL districts considered to be at-risk in 1990.
Two of these districts were more at-risk two years later.
These figures suggest that CL districts which were not considered at-risk initially, grew relatively more at-risk
between 1990 and 1992, while CL districts which were considered at-risk grew less so over the same period. These
results are not in and of themselves surprising. Ongoing research performed by the ADE R&D staff on factors
affecting student achievement suggest that it may be easier to improve the overall achievement levels of student
populations which are considered to be relatively more disadvantaged compared to those that are not.25 That is, it
may be more difficult to raise test scores in districts where student achievement levels are already high than to
improve the scores in districts where students perform substantially below average.
25 These remarks are based on preliminary results obtained fiom statistically modeling district aggregate
achievement levels using U.S. Bureau of the Census, at-risk and other related student and district characteristics.
Based on incremental changes in the explanatory variables, model elasticities indicate that larger marginal
increases in aggregate test scores are observed for districts displaying higher at-risk and lower socioeconomic
status. Model documentation is available upon request fiom the Research and Development Division, Arizona
Department of Education, 1535 W. Jefferson, Phoenix, Arizona 85007, or call (602) 542-503 1.
Appendix
Supplemental Tables
Weighted Student Counts for Group 1 Career Ladder Districts
1990 ITBS Complete Composite Record Information File
Research and Development, Arizona Department of Education
v
Amphitheater
Apache Junction
Catalina Foothills
Cave Creek
Creighton
Dysart
Flowing Wells
Ganado
Ky rene
Litchfield
Mesa
Peoria
Sunnyside
Window Rock
Total K-6
Studenb
4,827
1,884
1,328
562
3,472
1,517
1,819
Group 1 K-6 Non-Group 1
Students Percentlszutmm
1,162 24.07% 3,665
1,198 63.59% 686
772 58.13% 556
516 91.81 % 46
563 16.22% 2,909
279 18.39% 1,238
418 22.98% 1,401
Classroom data not available
3,527 46.62% 4,039
286 26.70% 785
1 1,005 23.84% 35,153
2,371 24.88% 7,158
2,287 42.41 % 3,106
33.5 24.82% Q54
Percent
75.93%
36.41 %
41.87%
8.19%
83.78%
81.61 %
77.02%
Total 86,395 24,699 28.59% 61,696 71.41 %
Note: The information presented above represents weighted average student counts compiled fiom 1990 Iowa Tests
of Basic Skills (ITBS) computer data files maintained by the Arizona Department of Education (ADE) Research and
Development Division. The figures presented were used to generate weighted average normal curve equivalent
(NCE) scores for Group 1 Career Ladder Districts. Missing student identifier and/or achievement scores may result
in inconsistent comparisons with enrollment, average daily membership and/or average daily attendance data
reported on the ADE School Year 1990 Year End Enrollment Report. In addition, missing data points which result in
the exclusion of student records fiom statistical operations may also be cause for inconsistent comparisons.
Group 1 Career Ladder Districts
Teacher Participation Rates
Percent of Total Teachers - All Grades
CTD Code District Name
100210 AMPHITHEATER
110243 APACHE JUNCTION
1002 16 CATALMA FOOTHILL
70293 CAVE CREEK
704 14 CREIGHTON
70289 DYSART
100208 FLOWING WELLS
10220 GANADO
70428 KYRENE
70479 LITCHFIELD
70204 MESA
7021 1 PEORIA
100212 SUNNYSIDE
10208 WINDOW ROCK
CTD Code District Name
1002 10 AMPHITHEATER
1 10243 APACHE JUNCTION
1002 16 CATALMA FOOTHILL
70293 CAVE CREEK
704 14 CREIGHTON
70289 DYSART
100208 FLOWING WELLS
10220 GANADO
70428 KYRENE
70479 LITCHFIELD
70204 MESA
702 1 1 PEORIA
1002 12 SUNNYSIDE
10208 WINDOW ROCK
Percent of Teachers Eligible for Caner Ladder Program
Descriptive Statistics and Estimated Parameters
Career Ladder Test Score Model
-Model Variables COIBLETB COXPOSITB - WgI- AVERME NCE SCORES, 1990 ITBS
(expressed as NCE points)
ABSK#TPP RATE (expressed in standardized units)
-IAN FAllILY INCOME, 1990 CENSUS (expressed in dollars)
LWTIO P#OLISH PROFICIENT (expressed in standardized units)
Percent of students in AND ILKDUCED PRICE LUWCH program
(expressed in standardized units)
-IAN HOUSING VALW: OWNER OCCUPIED HOUSING UNITS, 1990
CENSUS (expressed in dollars)
PXRCENT OF STUDENTS WITH CONPUTER IN HOME (1993 ASAP FORM
Dl, expressed as a percent)
PXRCENT NON-WRITE XNROLLl6lWT, K-12, 1991 (expressed as a
percent
Total number of Ca.08 in model r 140
Correlation Matrix of Model Variables with 1-tailed Significance Levels
Estimated Regression Parameters
Iotimation Mothod: O r d i n a r y L e a s t S q u a r e s
Dopondont Variable: CNCE-1 (COMPLETE COMPOSITE NCE SCORE, GRADES K - 6)
Nultiplm R .90489
R Squaro .a1882
Mjwtod R Squaro .a0921
Sturdard Error 38.49167
Analysis of Variance
_PL - lhLm4ua
Rogromsion 7 883856.02398 126265.14628
Rosidual 13 2 195572.39100 1481.60902
......................... Variables in the E q u a t i o n ..........................
V.ri.bl. -h
PmNwHT -87.765355
ABSENTEE -18.272694
INCOME .001641
LEP -20.872280
LUNCH -1.570439
NTI7-1 .624389
MVOOH 4.185843-04
(Constant) 391.603558
Cond .
mha
1.000
1.395
2.339
3.426
6.482
10.511
13.730
16.954
Collinearity Diagnostics
, - - - - Variurco Proportions -
mmr6 LJIl! LSmz
.00108 .00006 .00036
.00006 .05548 .04174
.00001 .04793 .00305
.00003 .54220 .30636
.00154 .21529 .51774
.02288 .03382 .00209
.07431 .04278 .lo626
.90008 .06244 .02242
Select Bibliography
Arizona Department of Education. Appendix to the Statewide Report for Arizona Pupil Achievement Testing.
Phoenix: Arizona Department of Education, June 1984.
Arizona Department of Education. Appendix to the Statewide Report for Arizona Pupil Achievement Testing.
Phoenix: Arizona Department of Education, June 1986.
Arizona Department of Education. Arizona Career Ladder Directory 1992-93. Phoenix: Arizona Department of
Education, 1993.
Arizona Department of Education. March 1993 Assessment Results, State of Arizona, Arizona Student Assessment
Program. Phoenix: Arizona Department of Education, September 1993.
Arizona Department of Education Research & Development Division. EdSTAE The At-Risk Status of Arizona
School Districts. Phoenix: Arizona Department of Education, March 1992.
Arizona Department of Education School Finance Unit. Class of 1992 Graduation Rate Study: A SIudy of
Graduation Rates for Arizona Public High Schoob by Caryn R Shoemaker. Phoenix: Arizona Department of
Education, March 1993.
Arizona Department of Education School Finance Unit. Dropout Rate Study 1991-92: A Study of Annual Dropout
Rates in Arizona Public Schools, Grodes Seven Through Twelve, by Caryn R Shoemaker. Phoenix: Arizona
Department of Education, February 1993.
Braver, Mary Walton, and Gerald C. Helmstadter. "Final Report on the Impact of the Arizona Career Ladder Pilot
on Student Achievement: Focus on Aggregated Analysis." Paper presented at the Joint Conference of the Arizona
Educational Research Organization and the Rocky Mountain Educational Research Association at Arizona State
University College of Education, Tempe, November 1990.
Braver, Mary Walton, and Gerald C. Helmstadter. "Executive Summary: Impact of the Arizona Career Ladder
Pilot on Student Achievement." Paper presented at the Arizona Legislature, Senate Education Committee,
Phoenix, February 1990.
Datasphere, Inc. Arizona Career Luabkr Program Evaluation: Macro Analysis. Phoenix: Datasphere, Inc., 1993.
Fuller, Linda L. "Arizona Career Ladder Program Evaluation Executive Summary." Paper presented at Career
Ladder Advisory Committee to State Board of Education, Phoenix, October 1993.
Fuller, Linda L. "Restructuring Education Systems: Key Organizational Factors Affecting Teacher Development
and Motivation." Ed.D diis., Northern Arizona University, Flagstaff, 1991.
Packard, Richard D., and Mary I. Dereshiwsky. Final Quantitative Assessment of the Arirona Career Ladder
Pilot-Test Project, Results of the Perceptual Assessment Scale, District Responses Reported by Survey Items, Sub-
Sections and Demographics. Flagstaff: Northern Arizona University Center for Excellence in Education, January
1991.
Packard, Richard D., and Mary I. Dereshiwsky. Summative Report VIII, Final Accumulative Results & Transfer of
Knowledge of the Arizona Career Ladder Research & Evaluation Project, Impact on Student Achievement,
Formulated Models, Network Anecdotes, & Recommendations to the Legislature for Policy Development,
Program Continuation & State- Wide Expansion. Flagstaff: Northern Arizona University Center for Excellence in
Education, January 1990.
Riverside Publishing Company. Iowa Tests of Basic Skills, Primary Battery, 1984-85 vs. 1987-88 National Norms
Comparison, Dtperences in Pupil PR, (conversion tables by grade) Chicago: Riverside Publishing Company, n.d.
Riverside Publishing Company. Iowa Tests of Basic Skills, Teacher's Guide, Multilevel Battery, Levels 7-8, Forms
G/H. Chicago: Riverside Publishing Company, 1986.
Riverside Publishing Company. Iowa Tests of Basic Skills, Teacher's Guide, Multilevel Batteiy, Levels 9-14,
Forms G/H. Chicago: Riverside Publishing Company, 1986.
Riverside Publishing Company. Iowa Tests of Basic Skilk, Teacher's Guide with I988 Norms, Levels 7-8, Form J.
Chicago: Riverside Publishing Company, 1990.
Riverside Publishing Company. Iowa Tests of Basic Skills, Teacher's Guide with 1988 Norms, Levels 9-14, Form
J Chicago: Riverside Publishing Company, 1990.
Riverside Publishing Company. Tests of Achievement and Proficiency, Teacher's Guide, Multilevel Battery, Levels
15-18, Fonn G/H. Chicago: Riverside Publishing Company, 1990.
Riverside Publishing Company. Tests of Achievement and Proficiency, Teacher's Guide with 1988 Norms, Levels
15-18, Form J Chicago: Riverside Publishing Company, 1990.
Select Statistical References
I Allen, Mary J., and Wendy M. Yen. Introduction to Measurement Theory. Monterey: Brooks/Cole Publishing
Company, 1979.
Johnston, J. fionometric Methods. 3d ed. New YO*: McOraw-Hill Book Company, 1984.
Klugh, Henry E. Statistics: The Essentials for Research. 3d ed. Hillsdale, New Jersey: Lawrence Erlbaum
Associates, 1986.
Rossi, Peter H., James D. Wright, and Andy B. Anderson. Handbook of Survey Research San Diego: Academic
Press, Inc., 1983.