Educational reform in Arizona: 1985-1990; a chronology of research & evaluation procedures for assessment of the pilot-test career ladders teacher performance & incentive programs

              A CHRONOLOGY OF RESEARCH & EVALUATION PROCEDURES
FOR ASSESSMENT OF THE PILOT-TEST CAREER LADDERS
TEACHER PERFORMANCE & INCENTIVE PROGRAMS:
1985 - 1990
by
Dr. Richard D. Packard, Manager
The Arizona Career Ladders Research & Evaluation Project
&
Dr. Mary I. Dereshiwsky, Research Associate
The Center for Excellence in Education 1 Research & Development Partnership
P. 0. Box 5774, Northern Arizona University
Flagstaff, Arizona 86011
Presented to the Joint Legislative Committee on Career Ladders Task Force
State Capitol, House Wing
July 20,1989
Table of Contents
Page
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Ove~ew. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1
1985 . Development and Planning for Program Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Origination of Evaluative Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Doctoral Dissertations and the Data Base . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4
1986 . Application of Research and Evaluation. Instrumentation.
Methodology. Reporting Procedures and Dissemination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Related Literature Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
PolicyResearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
The Perception Assessment Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Census Survey. Analysis and Reports of Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1987 . Expansion and Dissemination of Data Base Information . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Expansion of the Data Base . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7
Information Dissemination and Reporting of Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1988 . The Emergence of a Predictive Model for Effective
Career Ladders Program Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9
Directional Focus of the Research and Evaluation Project . . . . . . . . . . . . . . . . . . . . . . . . . 9
The Career Ladders Pilot-Test Program Network ...............................1 0
Individual School District Research and Dissemination . . . . . . . . . . . . . . . . . . . . . . . . . .1 0
Instrumentation Reliability and Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1989 .A dvancement in Project Focus. Research Design and Methodology . . . . . . . . . . . . . . . . . . . . 12
The Focus of Teacher Performance and Student Achievement . . . . . . . . . . . . . . . . . . . . . . . 12
Individualized Assessment Through Case Study Methodology . . . . . . . . . . . . . . . . . . . . . . 13
Convergence and Profiling from Quantitative and Qualitative Data . . . . . . . . . . . . . . . . . .1 3
Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1.4
Researchinprogress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1.5
Planning Documentation into the 1990's . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15
Summary and Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1.6
A Potential Breakthrough . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16
Epilogue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1.8
List of Figures
Figure 1: Research and Evaluation Project Procedure for Policy Development and Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Figure 2: Mapping the Developmental Process for Effective School Reform: Designs and Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 9
Figure 3: District Readiness Profile of Strengths and Weaknesses .S upport . . . . . . . . . . . . . . . . . . 20
Figure 4: District Readiness Profile of Strengths and Weaknesses .F ocus . . . . . . . . . . . . . . . . . . . 21
A CHRONOLOGY OF RESEARCH AND EVALUATION OF THE PILOT-TEST CAREER LADDERS
TEACHER PERFORMANCE AND INCENTIVE PROGRAM
1985-1990
Overview
In 1985 the Center for Excellence in Education at Northern Arizona University responded to the
opportunity to evaluate the probability of program success of the career ladder pilot-test project initiated by
the passage of S.B. 1336. In addition to yearly research reports on the current status of the program, the
objectives of the Arizona Career Ladder Research and Evaluation Project have expanded to include making
recommendations as to individual districts' readiness to support major change and reform programs. The
purpose of this report is to trace and summarize the activities of the research and evaluation process related
to the five-year pilot-test project, from its legislated beginning through the present time, including
forthcoming activities. Each summary description will be followed by a bibliographic listing of related
research reports, presentations and publications.
Figure 1, page 2, presents a diagrammatic overview of the outputs of the research and evaluation
project. Reading from left to right, this model depicts the process by which data collected by the center is
systematically analyzed, summarized and formulated into specific policy recommendations to be made to the
Joint Legislative Committee on Career Ladders (JLCCL) at the end of the five-year pilot-test. This process
effectively serves as an overall framework for the more specific activities to be described in greater detail and
in time order below.
Two major findings have emerged to date which have influenced the direction of research and
evaluation activities. The first of these is a startling discrepancy in individual districts' readiness to support
change and reform movements such as career ladders, despite supposedly "uniform" legislative program
requirements. As a result, evaluation has shifted from overall monitoring of compliance of these
Dr. Richard D. Packard, 1989
Figure 1
I. DATA BASE
DOCUMENTS
INCLUDE A
SERIES OF
QUALITATIVE
AND
QUANTITATNE
PLANS,
DISSERTA-TIONS,
ANALYTIC
STUDIES AND
PUBLICATIONS
1I.SUMMARY
EVALUATION
DOCUMENTS
ARE
DESCRIPTIONS
AND LISTINGS
OF
EVALUATIVE
JUDGMENTS
BASED ON
QUANTrI'ATIVE
AND
Q U r n A V E
RESEARCH
METHODOLOGY
111. MODELS.
DESIGNS
AND
STRUCTURES
INCLUDES
CONcErnAL
MODELS
WHICH
DIAGRAM-MATICALLY
s-
PROGRAM
COMPONENTS
AND
PROCESSES
IV .POLICY
DOCUMENTS
INCLUDE
ESSENTIAL
REQUIREMEN?
FOR
LEGISLATION
AND
PROJECTED
EFFECTS OF
NON-LEGISLATION
ON REFORM
guidelines, to include careful assessment of the cwrent operating status of support and focus factors for
change within districts on an individual basis.
The second conclusion apparent from the first five years of program research is an unrealistic
expectation which is prevalent with respect to the change process in the social sciences. There has been a
disturbing tendency for adopters of programs such as Career Ladders to simply "attach them to existing
organizational structures and expect both automatic integration and instantaneously successful outcomes.
What these well-intentioned innovators fail to realize, however, is that new discoveries in the social and
behavioral sciences are as evolutionary and developmental in nature as their counterparts in the medical,
physical and natural sciences. Processes for change and improvement in these fields require a carefully
planned start-up time, as well as continual monitoring, feedback and adaptation to changing conditions.
Therefore, a careful understanding of the elements of the change process, as well as a long-run strategic plan
for evaluating the resulting outcomes, are necessary for a complete and valid evaluation. Research efforts
during the five-year pilot-test period have incorporated an identification and application of such a model to
the change and reform process in education.
1985 - Develo~menat nd Planning for Propram Evaluation
Evaluation Desi~n
Arizona legislation established the Center for Excellence in Education at Northern Arizona
University as the designated evaluator of the "Career Ladders Pilot-Test Teacher Performance and Incentive
Program." An evaluation design was formulated and approved, which consisted of an "improvement
model" requiring districts to follow a yearly formative and cyclical planning, implementation and
sumrnative evaluation process. Along with evaluation, the Center was to provide monitoring of important
legislated guidelines.
Origination of Related Evaluative Com~onen~
In 1984-85, the researcher culminated studies relating to basic human psychological needs and
organizational functions which result in the greatest worker motivation and production. Several factors
were identified as being instrumental to successful change efforts. These included the quantity and quality of
worker competencies, as well as their performance in meeting the objectives of public and private
organizations. These research results were later incorporated into evaluative components of individual
districts' career ladder plans. Their primary objective was to assess the districts' ability to provide adequate
incentives for superior teacher performance. A second aspect of the evaluation process was districts' ability
to motivate teachers to improve their skills and expand their responsibilities. This research later became a
part of comparative studies which determine the relationship between a district's "organizational climate" or
"psychological environment" and career ladders (CL) program success. A second objective was to evaluate
trend data collected during the pilot-test period in order to determine the direction and extent of impact which
the CL incentive plan has had on interpersonal communication within each organization.
Doctoral Dissertations and the Data Base
An important part of legislative policy intent was to involve university students more actively in
various phases of the evaluation process. The first doctoral dissertation related to Career Ladders was
conducted at Northern Arizona University. Recent technological advances with respect to the study
variables, as well as the complex and broad-based nature of this evaluation process, have resulted in ongoing
research efforts at all three state universities. The popular effective schools movement, for example, has
been the focus of a number of current papers, theses, and dissertations. Specific areas being investigated
include CL program goals of teacher improvement, administrative leadership, and enhanced accountability
for student achievement.
Packard, R. D. (1985). Arizona career ladders e v a l d e s i g n . Document on
evaluation research design presented to the Arizona Joint Legislative Subcommittee on
Career Ladders, State Capitol, House Wing, Phoenix, Arizona.
Packard, R. D. (1985, August). Determining administrative effectiveness in manaping human
resources for develo~ment and Droductivitv. Paper presented at the Resources
Management Symposium, Northern Arizona University, Flagstaff, Arizona.
Packard, R. D., & Bierlein, L. (1985, December). develop in^ and implementing a career ladders
evaluation process: The Arizona e.Docum ent distributed by the National
Association of Secondary School Principals, Washington, D. C., to the participants of
the Inservice Program for Administrators Conference, Tucson, Arizona.
One doctoral study initiated continuing research and analysis of related literature. It involved an
extensive review of such key areas as "differentiated staffing" and "incentive plans." In addition, it
investigated the effects of other, more specific variables related to the many complex programmatic issues
of concern. These included the following: teacher and administrator evaluation, inservice development,
program input and ownership, change, reform, motivation, communication, leadership, organizational
management, support systems, curriculum, student achievement, and testing.
An especially important objective of the literature review was identifying the reasons for the
success or failure of previous educational reform movements. This focus was necessary, in order for the
evaluators to identify those specific factors which would be most indicative of district "readiness" levels for
successful program implementation.
Policy Research
The initial interest of outside agencies and research centers in the five-year pilot-test eventually led
to an ongoing association with other social and political entities. Continued input from these agencies has
extensively influenced research purpose and methodology of the "Arizona Project." Such active
involvement has helped to determine the potential for legislative bodies to use "policy research" or
"evaluation research results in making legislative or policy decisions.
The Percmtion Assessment Scale
The primary career ladders survey instrumentation, entitled the Perce~tionA ssessment U,was
designed in order to elicit information on attitudes and perceptions of organizational functions. This survey
was developed by a research team composed of members from the three state universities and legislative
research analysts, and it was initially pilot-tested in two CL districts. The instrumentation included forced-choice
and open-ended questions related to CL concepts and organizational climate, resulting in both
quantitative and qualitative data. (Three years of data have accumulated from the scale which will allow for
trend analyses.)
Census Survev. Analvsis and Reports of Findin=
The total population of "Phase In school districts was surveyed, data were analyzed and initial
findings were presented to the JLCCL. (Phase I1 and 111 districts were subsequently evaluated as they
implemented their programs.) More specific and detailed data reports were provided to each school district
for analysis and improvement of individual areas of weakness. The intent has been for each district to
evaluate the findings and implement the initial improvement model of refinement and recycling, based on
research evidence. As a result, change and reform would be based on input from teachers and other
individuals who have a vested interest in improving themselves and the achievement of their students.
Packard, R. D. (1986, January). Organlzatlonal scale . . . Copyright
Registration Number-TXu 226 869, United States Copyright Office, The Library of
Congress, Washington, D. C.
Packard, R. D. (1986, January). The Arizona career ladders research and evaluation ~roiect
jmplementation plans. Drocedures and assi~nments, Paper presented to the first meeting
of the combined pilot-school districts, Grand Canyon College, Phoenix, Arizona.
Packard, R. D., & Bierlein, L. (1986, February). O r g m u a m d
. .
Performance Assessment Scale. Snowflake Unified School District, Snowflake, Arizona
Packard, R. D., Bierlein, L., Aleamoni, L., & Helmstadter, G. C. (1986, March). Perceptian
AssessmentScRaelseear.c h instrumentation developed by the Arizona Career Ladders
Research & Evaluation Team for the assessment of the Arizona Career Ladder Teacher
Incentive Programs, State Capitol, Senate Wing, Phoenix, Arizona
Packard, R. D. (1986, April). nirteen ~svcholoeical factors in assessment of svstem
effectiveness in meetine &&y develo~ment needs. Paper presented to the National
Conference on Faculty Evaluation and Development: Lessons Learned, Kansas City,
Missouri.
Packard, R. D., Kundin, K., & Bierlein, L. (1986, April). Pilot for the A&?nuua
-. The Kachina School, Peoria School District, Peoria,
Arizona.
Packard, R. D. (1986, May). A description and proiection of the effects of the Center for
Excellence in Education as it relm to career ladder teacher incentive promarns on state
and national educational pow. Paper presented to The Rand Corporation in consortium
with Rutgers University, The Eagleton Institute of Politics, and Wisconsin Center for
Education Research, five year study on policy research, Phoenix, Arizona.
Packard, mR. D. (19.8P6,a pJuern pe)r.e seTneteda toc thhee Wrin ddow Rock Schooel District Steerivng Comrniaee one
Teacher Development, Center for Excellence in Education, Northern Arizona University,
Flagstaff, Arizona.
Packard, R. D. (1986, July). Jmplications of career ladder teacher incentive proyrama. Paper
presented at the 16th Resources Management Workshop, Northern Arizona University,
Flagstaff, Arizona.
Packard, R. D., Pavlich, P., & Bierlein, L. (1986, July). 9
evaluation Document prepared for general dissemination to education
and other public sources, University News and Publications, Northern Arizona
University, Flagstaff, Arizona.
Packard, R. D. (1986, August). P c hand ve- Paper
presented to the faculty of the Center for Excellence in Education Workshop, Northern
Arizona University, Flagstaff, Arizona.
Packard, R. D. (1986, Fall). Essential elements for assessment of system effectiveness in
meeting faculty development needs. &, Kansas State
University, Manhattan, Kansas.
Packard, R. D. (1986, November). A s m~ilot te ac . . her incentive program: Research and
-merit for whcv change and reform. Paper presented at the National Council of
States, Eleventh Annual National Conference, Hyatt Regency Hotel, Nashville,
Tennessee. (ERIC Document Reproduction Service No. ED 275 654.)
Packard, R. D., & Bierlein, L. (1986, December). the mR.eport to the Joint Legislative Committee on
Career Ladders, 1st Annual Research Report, Arizona House Wing, State Capital,
Phoenix, Arizona.
1987 - Expansion and Dissemination of Data Base Information
Expansion of the Data Base
A dissertation related to performance-based teacher evaluation expanded the review of related
literature and analyzed quantitative and qualitative data through basic descriptive, anecdotal and non-parametric
statistical procedures. As a result of the relationship of success of reform movements and level
of organizational operations, effective schools research was reviewed and more closely associated with CL
research findings. In order to satisfy one of the most important concerns of legislative intent, plans were
initiated which related to the scientific study of accounting for program effects on student achievement.
. . .Information Dissemn-
Considerable outside interest in the research results of the career ladders pilot-test program became
evident during this time. This was a direct result of the focus on relevant issues such as teacher
development and incentives, as well as their effect upon student achievement Several state, national and
international presentations related to topics such as the following: a) teacher productivity, performance
evaluation and rewards; b) program designs and structures; c) reform and change in education; and d) teacher
productivity and instructional leadership roles.
A major research finding during this time concerned the extreme diversity of individual districts'
readiness to implement a uniform set of legislative guidelines. This diversity in readiness levels existed
despite the otherwise carefully matched similarities in CL program goals, designs and structures across
participating districts. As a result, reporting of project results to the JLCCL became considerably more
complex. Quantitative and qualitative data indicated that factors & than CL plans and goals were, in fact,
instrumental to potential program success or failure.
Documentation
Packard, R. D., & Bierlein, L. (1987, January). Arizona weer ladder reswch and evaluation
ct: Research and d e v e l w n t for effective e d d o n a l chance and reform. Ann
Arbor: The University of Michigan, Counseling and Personnel Services Clearing House,
2108 School of Education (ERIC Document Reproduction Service No. CG 019 609.)
Bierlein, L. A. (1987, March).
cts toward ~ e r f o r m a n c e - b a s e d o n ~rocedures.
Unpublished doctoral dissertation, Northern Arizona University, Flagstaff, Arizona.
Packard, R. D., & Bierlein, L. (1987, JundJuly). "Career ladder facts abstract and incentive
programs for teachers: Will it work in Arizona?" Arizona Administrator Journal, 16
(No. 9), 9-10. (ERIC Document Reproduction Service No. ED 287 797.)
Packard, R. D. (1987, July). Devel- of e d u c a t i o n a l FoWmg indtv~dual . . , .
of -vvti . . and in e m . Pa per
presented at the World Assembly of the International Council on Education for Teaching,
Eindhoven, The Netherlands. (ERIC Document Reproduction Service No. ED 283 793.)
Packard, R. D. (1987, October). &arch & evaluation: 1987 ~reliminawre mrt for the car=
ladder teacher incentive and develo~ment vr-. Document presented to the Joint
Legislative Committee on Career Ladders, Arizona State Capitol, House Wing, Phoenix,
Arizona.
Packard, R. D. (1987, October). r l r
pro1 ec t re port. Document presented to the Joint Legislative Committee on Career
Ladders, Arizona State Capitol, House Wing, Phoenix, Arizona.
Packard, vR. D.. & .F argo, S. (1987, Nove.mRbeesr)e.a r9ch docum ent presented to the Joint Legislative
Committee on Career Ladder Programs, Arizona State Capitol, House Wing, Phoenix,
Arizona.
. . Packard, R. D., Dereshiwsky, M., & Groenendal, J. (1987, November). kimptlve & analyhl w.Research docum ent presented to the Joint
Legislative Committee on Career Ladder Programs, Arizona State Capitol, House Wing,
Phoenix, h n a . (ERIC Document Reproduction Service No. SP 029 861.)
Packard, R. D. (1987, November). Research & ev-n: 1987 results for a
dev-. Paper presented at the National
Council of States, Twelfth Annual National Conference, San Diego, California. (ERIC
Document Reproduction Service No. ED 288 878.)
. .
Packard, R. D. (1987, November). North evaluation of effective sr;haals
m. Document presented to Thunderbird High School, Glendale School District,
Phoenix, Arizona.
Packard, R. D. (1987, November). Qutline of similarities & diversities in achieving performance
@wets & conditions of ~ i l ote st career ladder Droprams in the state of Arizona.
Document presented to the Joint Legislative Committee on Career Ladders, Arizona State
Capitol, House Wing, Phoenix, Arizona.
. . Packard, R. D., & Nichols, W. (1987, November). Qu&atrve a&sk&m& for the 1987
& weaknesses. Research document presented
to the Joint Legislative Committee on Career Ladder Programs, Arizona State Capitol,
House Wing, Phoenix, Arizona.
Packard, R. D., & Momson, L. (1987). Analysis of the initial Arizona career ladder teacher
incentive programs. Excellence In T- 5 (No. I), 4-6. (ERIC Document
Reproduction Service No. ED 277 687.)
Helmstadter, G. C. (1987). 9. . . . .
vP.aper presented to the Arizona Joint Legislative Committee on
Career Ladders, Arizona State. University, Tempe, Arizona.
1988 - The EIBZIWXX of a Pre&c@veM ode
. . 1 for Effective
r Ladders P r o m Imlementabon
Research findings during the preceding year had revealed a dichotomy with respect to CL program
implementation of districts which were otherwise similar in terms of goals and models for designs and
structures. That is, some of these seemingly equivalent districts were successfully reaching their target
objectives, while others were failing to make observable progress. As a result, research efforts in 1989
were focused on identifying and assessing the primaty reasons for these dichotomous results.
The extreme diversities in readiness found in the CL research, combined with private and public
organizational and effective schools research, identified those operational factors within the districts which
influenced performance and production. This led to the emergence and continual refinement of an integrated
model for assessing readiness (see Figure 2, on page 19) of districts for successfully implementing the CL
PK'gram.
The Career Ladders Pilot-Test Program Network
The Network is an association of the CL school districts which developed primarily as an
important support for project research and evaluation. Extensive time and effort has been expended by this
committee and its research sub-committee in assisting the researchers with evaluation content and collection
of data
Several career ladders teacher-leaders, adminisaators and doctoral students within districts initiated
program research, writing and dissemination of findings in cooperation with the project evaluators at this
time. A number of specific components of the model have been enhanced through in-depth study of topics
such as the following: a) production and outcomes in student achievement; b) change theory; c) program
designs and structures; d) development of district R & D; e) teacher development, leadership, mentoring,
peer instructional coaching and clinical supervision; and f) cooperation with the development of a predictive
achievement model. In addition, requests for information from non-career ladders districts throughout the
state resulted in wider dissemination of CL fmdings to date.
o-Rn . .. Vali&y
The survey items were evaluated with respect to response consistency
through calculation of standard reliability indices. A factor analysis was also conducted within each
subsection in order to identify clusters of questions which best defined each topic area.
Documentation
Packard, R. D. (1988, January). Pesearch auestions for the development of comoonents of an
smer~ineca reer ladder program m U . Docurnent presented to the Career Ladder Pilot
Network, Mesa Public Schools Administration Center, Mesa, Arizona
Packard, R. D. (1988, March). u a t i v e~Dlnni . . ng.
Document presented to the Joint Legislative Committee on Career Ladders Research Staff
and the "Network," Mesa Public Schools Administration Center, Mesa, Arizona.
Dereshiwsky, M. I., & Packard, R. D. (1988, April). Predictive achievement model. Paper
presented to the Career Ladder Steering Committee, Sunnyside School District, Tucson,
Arizona.
Packard,m R. .D .. (1988, April). Research RevaluDatoi.ocnu:m oef ntth per enseetnwteo drk t ot tuhek Shtatxe
Career Ladder Task Force Research Committee.
Packard, R. D., & Dereshiwsky, M. I. (1988, April 30). Evaluation Studv of
ofa &
instudent Research document presented at the Arizona Educational
Research Association for the Conference on Partnerships in Education, Tucson, Arizona.
Karp, S. (1988, May). The effects of change on school climate: A case study of a career ladder
innovation. Unpublished doctoral dissertation, Northern Arizona University, Flagstaff,
Arizona. (Cropper, Dissertation Chair.)
Packard, R. D. (1988, Fall). "Career ladders: One form of educational reform."
Teaching, 6 (NO. I), 4-7.
Packard, R. D., & Dereshiwsky, M. I. (1988, October) 0
&x&&mm. Document presented at the Arizona Education Association Leaders'
Conference, Hyatt Regency, Phoenix, Arizona.
Packard, R. D., & Dereshiwsky, M. I. (1988, October) -ram designs and structures.
Document presented to the Joint Legislative Committee on Career Ladders, Arizona State
Capitol, House Wing, Phoenix, Arizona. (ERIC Document Reproduction Service No.
ED 302 534.)
Packard, R. D., Dereshiwsky, M. I., Gonzales, M., & Fimbres, E. (1988, November). Research
:-r~ve Evaluating and local -act on
&v-ved student. Paper presented to the
National Council of States, New Orleans, Louisiana. (ERIC Document Reproduction
Service No. ED 302 532.)
Packard, R. D., Dereshiwsky, M. I., & BasIIsaac, E. (1988, November) -rated model for
the ~rofessionald evelopment of teacher leaders. Paper presented to the National Council
of States, New Orleans, Louisiana. (ERIC Document Reproduction Service No. ED 302
531.)
. . Packard, R. D., & Dereshiwsky, M. I. (1988, November 29). Qjwwatw levels of pmgr.an
v careerlDoacumdent prdesenteed tor the Joint Legislative
Committee on Career Ladders, Arizona State Capitol, House Wing, Phoenix, Arizona.
(ERIC Document Reproduction Service No. ED 302 530.)
Packard, R. D., & Dereshiwsky, M. 1. (1988, November 29) Qur&&tt~vle . . vels of Drogram
gcce~tabilitb~y corn-ts and demoera~hicc onditions. Document presented to the
Joint Legislative Committee on Career Ladders, Arizona State Capitol, House Wing,
Phoenix, Arizona. (ERIC Document Reproduction Service No. ED 302 535.)
Packard, R. D., & Dereshiwsky, M. I. (1988, November 29) A v ete&
for enhanced.Do cument
presented to the Joint Legislative Committee on Career Ladders, Arizona State Capitol,
House Wing, Phoenix, Arizona. (ERIC Document Reproduction Service No. ED 302
533.)
Packard, R. D., & Gonzales, M. (1988, December 12). and Prar;edures f o ~
lA series of studies presented t.o the Crane
School Dismct, Yuma, Arizona.
Kundin, K. R. (1988). The wP:erceptisms of educatauiud
vear two w.Unpu blished doctoral dissertation, Arizona
State University, Tempe, Arizona.
Packard, R. D., & Dereshiwsky, M. I. (1988). A reliabilitv assessment of the subscales of the
1988 perce~tiona ssessment scale survey.
Packard, R. D., & Dereshiwsky, M. I. (1988). A factor analvsis of the subscales of the 1988
perce~tiona ssessment scale survey.
Packard, R. D., & Dereshiwsky, M. I. (1988). Demo gera-Dhic testing:
1988 survey.
A lack of knowledge and application of adequate procedures related to measurement science and
technology became apparent as a result of the presumed legislative intent to connect teacher performance to
student achievement. School districts across the country continue to make the assumption that when
teachers are evaluated on their instructional processes, there is automatically a valid connection to student
achievement. More advanced research designs and models using multivariate statistical analysis procedures
are being applied within three dissertations at the three state universities. These quantitative methods are
more realistic, since they enable researchers to assess the joint effects of variables such as CL participation,
years of teaching experience, and grade level taught. In addition, multivariate methods allow for inclusion
of multiple associated measures of student pedormanCe, such as a standardized test and a teacher-made test.
Preliminary results indicate a potential breakthrough in support of initial legislative opinions that teachers
should be evaluated and rewarded based on levels of performance, rather than solely on years of experience
and college credit hours.
Jnd.~ v. idualizedA ssessment Thrqueh Case && Methodol~gy
Data originally collected through the Perception Assessment Scale survey identified the extreme
diversities of districts with respect to readiness levels of the support and focus factors of effective change
processes (see Figure 2 on page 19 for a depiction of the model). Additional research and data-analysis
procedures were necessary in order to provide a more comprehensive picture of the actual program
implementation processes operating within these individual districts. Through case study methodology the
researchers have piloted procedures which have provided valuable insights into the "operational functioning
levels" of key support and focus components. Focus-group interviews conducted with all relevant sub-groups
within a given system have provided an in-depth picture of participants' perceptions and feelings
concerning their CL program experiences. Matrix analysis procedures have been used to summarize the
open-ended responses to the Perception m s m e n t Sc& survey, in order to identify the most commonly
occurring areas of satisfaction and dissatisfaction with district- and school-level organizational climate.
Such clustering and comparative analysis of qualitative (non-numeric) data represent the latest methodologic
advances for understanding and reporting key participants' experiences in their own words. The result is a
more valid, complete and rich data base, adding to existing knowledge in the social and behavioral sciences.
. . . . Converpence and Profih e- fuOrom
The foregoing variety of quantitative and qualitative data-analysis procedures, as applied to policy
and evaluation research issues, represent the current state-of-the-art. The results of applying such multiple
and diverse methods as survey, case-study and model-validation techniques are continually assessed for
consistency in order to determine the extent of methods convergence. A high degree of methods
convergence means that the application of a number of different analytic techniques and data collection
procedures has resulted in similar answers to the research question under study. This effectively indicates
cross-validation, allowing for a corresponding high level of confidence in the conclusions.
An especially promising avenue of current assessment research involves the development of a
profile for each district, based on specific quantitative and qualitative data resulting from the evaluation
process. (Figures 3 and 4, pages 20 and 21, contain examples of such profiles.) Individual components
(support and focus factors) of the readiness model are graphically depicted in terms of their cumulative
frequency of mention as being positive or negative. That is, the position of each factor relative to the
baseline of zero indicates its direction and magnitude of impact upon district readiness for change. By
examining the components of these profiles, one can identify at a glance those support and focus factors
which are operating at satisfactory levels, and which are in need of remediation. The primary finding
depicted in this profile is that those factors which should be supportive of teacher and student progress have
instead turned out to have a negative impact.
A number of other research studies are currently underway; these are listed in the "research in
progress" section. Topics being investigated include factors related to assessment of student achievement,
measurement of teacher effectiveness, and identification of attitudes and opinions of educational policy
leaders.
Documentation
Packard, R. D. (1989, FebruaryMarch). "Evaluating a Teacher Incentive Program." T h e m
wsletter. 1 (No. I), 3-4.
Packard, R. D., & Dereshiwsky, M. I. (1989, M1arc-h 30). of
d to school. Paper presented to the Far West Laboratory for
Educational Research and Development: Teacher Incentive Discussion Group, San
Francisco, California.
Packard, R. D., & Dereshiwsky, M. I. (1989, April 17). getails of proeram evaluation desicn
jmd form& Document presented to a rural Arizona career ladders school district.
Packard, R. D., &.Dereshiwsky, M. I. (1989, April 20). A study of essential components and
jevels of school e ffectiveness slrg-wrtin e_ career ladders roer ram develo~mentin the
Window Rock School D i s a . Paper presented to a mal Arizona career ladders school
district.
. . Packard, R. D. (1989, May 2). Models the deD end-enecv of
l levels of e s s e n t i a l 0 0 1 ef fectivmss. Paper
presented at the ASA Annual Conference, Division of Higher Education, Crescent Hotel,
Phoenix, Arizona.
Fritz, E. S. (1989, May). M e ein administrative evalu-grams to SUDDOG
effective fmltv devh~menitn c a . . r ladder d~strrcU~.n published doctoral dissertation,
Northern Arizona University, Flagstaff, Arizona. (Packard, Dissertation Chair.)
Thomas, J. L. (1989, May). The career ladder mt: Perc-s of teachers in
t o ~ D e r f o r m a n c e - b a s e d o c e ~ .
Unpublished doctoral dissertation, Northern Arizona University, Flagstaff, Arizona.
(Packard, Dissertation Chair.)
Packard, R. D., & Dereshiwsky, M. 1. (1989, June 9). Qumuauve analysis of the perc- . .
assessment. An assessment and evaluation of an educational reform program in a
rural Arizona career ladders school district.
Packard, R. D., & Dereshiwsky, M. I. (1989, June 9). A profile of strengths and insufficiencies
in district readiness for educational reform. An assessment and evaluation of an
educational reform program in a rural Arizona career ladders school district.
. . Packard, R. D., & Dereshiwsky, M. I. (1989, June 9). pAn.
assessment and evaluation of an educational reform program in a rural Arizona career
ladders school district.
Packard, R. D., & Dereshiwsky, M. I. (1989, June 9). Case research: A model for si-set
tin^. embedded, focus-P~OUinDte rview desien & analysis ~ r o c e d m .A n assessment
and evaluation of an educational reform program in a rural Arizona career ladders school
district.
Packard, R. D. (1989, Summer). Social conditions for change and reform: What is Arizona like?
Z0.S. Save Our Schools...Save Our &ate 11 - ". A report pending
publication for the Arizona Town Halls of the Arizona Academy, 54th Town Halls
Assembly, Prescott, Arizona, pp. 123-126.
Research in P~ML~SS
Planning Documentation into the 1990's
Bailey, L. (198911990). O r e l a t i o n s h i D lev els of s c h d
effectiveness. Doctoral dissertation in progress, Northern Arizona University, Flagstaff,
Arizona. (Staires, Dissertation Chair.)
Fimbres,e E. (198911990).. 0. Dcctora.l dissertation in progress, Northern
Arizona University, Flagstaff, Arizona.
Fuller, L. (198911990). Effects of -ut. r e. bia.s and motivation base on preanizational involvement in career ladder ~ilot-testa stncts. Doctoral dissertation in
progress, Northern Arizona University, Flagstaff, Arizona. (Packard, Dissertation Chair.)
. . Hunter, G. A. (1989119 9t0-). boar-ort of v e- proprams.
Doctoral dissertation in progress, Northern Arizona University, Flagstaff, Arizona.
(Packard, Dissertation Chair.)
. . Morrison, L. (198911990). U d vo f th nee-er o r ~ o n a c!lim ate and
gffective schools. Doctoral dissertation in progress, Northern Arizona University,
Flagstaff, Arizona. (Packard, Dissertation Chair.)
Nine, G. (1 98.91-1990). Doctoral dissertation in progress,
Northern Arizona University, Flagstaff, Arizona. (Packard, Dissertation Chair.)
Packard, R. D., & Dereshiwsky, M. I. (198911990). 0
survey. A
research document in progress.
Packard, R. D., & Dereshiwsky, M. 1. (1989J1990). 0. .
Bistrict. A research document in progress.
Santesteban, P. (1989190). T h e eof w ard educational
d to theDocitoral dissmertation in progress, Northern Arizona University, Flagstaff, Arizona. (Packard, Dissertation Chair.)
Santesteban, T. (1989190). The divergencelconvergence of policv leaders toward educational
reform related to oreanizational suvport factors. Doctoral dissertation in progress,
Northern Arizona University, Flagstaff, Arizona. (Packard, Dissertation Chair.)
A Potential Breakbugh
Research and evaluation of the pilot-test program over a period of time has resulted in the
following three major findings:
1. Successful change and reform can be influenced by intervention programs
such as career ladders, but it is primarily dependent upon well-developed
and effective school systems.
Development of effective schools includes the following key variables: a) competent and
supportive leadership; b) clear and positive communications and interpersonal relationships; c)
reliable and valid evaluation of teachers and administrators; d) accurate application of advanced
science and technology; e) curriculum and instruction which meets local student needs; and f)
reliable and valid accountability for student achievement.
2. The association between teacher performance and competency based on
process and developmental evaluation can scientifically be related to
reliable and valid student academic achievement measures.
Many organizational bureaucracies, including school systems, compensate personnel
based solely on years of experience and formal course work. They are not predominantly
rewarded based on competency and performance. Unfortunately, most districts are still
operating on the assumption that mere observation of teachers' instructional processes which
result in good teacher evaluations automatically imply a high rate of learning for their
students. The preferred alternative, which has not been adopted to the extent warranted, is to
examine the magnitude of student achievement gains. In most districts, more valid and
reliable methods of evaluating teacher performance and student achievement are needed.
3. While the Career Ladders Pilot-Test Program does an excellent job of
assuring teacher accountability, it has been far less successful with respect
to assessing the corresponding accountability of other key factors of
district operational effectiveness.
Teachers have effectively been held accountable in terms of time on task, expanded
responsibilities focused on instructional improvement, good classroom teaching and
communication skills and demonstrating student achievement gains. However, the program
has not held districts accountable for the development of an adequate curriculum, for
improvement in communications and governance operations (participatory management,
shared responsibilities, etc.) or for the development of long-range plans for demonstrating
student achievement gains in individual schools as well as across the total district. More
effective indicators of district-wide progress would enhance the evaluative aspects of the
program.
Review of Progress
From 1985 to the present, the career ladders pilot-test research and evaluation methodology has
evolved from a general evaluation design, to identification and description of relevant concepts and processes
and in-depth into a variety of data collection, analysis and reporting procedures. Since career ladders
program success was found to be dependent primarily upon factors other than the legislative and district
goals and district CL plans and structures, current research has centered on essential support and focus
factors which must be operating effectively before a teacher incentive and development program can
progress. It has become apparent that an effective school organization and operation is necessary in order
for all participating districts to be equally successful in carrying out the legislative intent of the CL policy.
Democracy and the Freedom to
Pursue Individualities
and Uniquenesses
Politically, in a democracy, individuals and groups,
with their uniquenesses and diversities, are equally
recognized in the pursuit of freedom and independence.
Socially and behaviorally, in a democracy, these
diversities must be recognized and allowed the
freedom to develop in the direction of their
uniquenesses and at the rate of their own potentialities
for growth.
Educationally, equal treatments or interventions
placed on unequal (unique) individuals, socially and
behaviorally, fail to recognize diversities and
uniquenesses and the way in which human beings and
organizations develop.
To make everyone developmentally equal would
require genetic and environmental interventions and
manipulation by powerful non-democratic forces; by
doing so, everyone would have an externally mandated
potential to become socially and behaviorally "equal."
Equal treatment in intervention and developmental
time, applied to unequal and unique entities, does not
democratically, socially or behaviorally recognize the
uniqueness and diversity of individuals or groups
within our free society.
There is no way to recover from being socially and
emotionally unequal, and in a democracy it is
definitely not necessary. All individuals must have
the freedom, independence and support to develop
their own potential.
Dr. Richard D. Packard. 1988
Figure 2
MAPPING THE DEVELOPMENTAL PROCESS
FOR EFFE(JTTvE SCHOOL REFORM:
Designs and Structures
INFLUENCE OF
UNIVERSITY ROLE,
RESPONSIBILITIES CQNDlTIONS ON
AND NEEDS FOR EDUCATION &
'IHE PRIVATE
LEGISIATZVE GUIDELINES
ANALYSIS OF RESEARCH VARIABLES,
PROGRAM EVALUATION & IMPROVEMENT
A Model of Interrelated Components of Program Support and Focus
for Effecting Change and Reform in Education
19
Figure 3
DE$TRUCT' READmBSS PROP- OF STIRBNGTIlJB AND WBAKNBSSBS
LOCAL PPNANCW I
-
OBNBW.
PINANCBI
Sufficiency of Readiness Baseline
0
PROFSSSIONAL
(OWNBRSHIP)
DISTRICT
R I D
BVALUATION
kid
B