Arizona validity and standard setting study of the pre-professional skills tests

              ZONA BOARD OF REGENTS
ARIZONA VALIDITY AND
STANDARD SETTING STUDY
OF THE
6. 6. Helmstadtor
Prodsrasar
Arizona State Untiverslty
Odua V. EIllott
A88oclate Director tor Academic Programs
Arizona Board of Regents
TABLE OF CONTENTS
Page
I n t r o d u c t i o n . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Background . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Description o f the PPST . . . . . . . . . . . . . . . . . . . . 1
Overview o f the Study . . . . . . . . . . . . . . . . . . . . . 2
Procedures Followed . . . . . . . . . . . . . . . . . . . . . . . . 3
Obtaining the Review Panelists . . . . . . . . . . . . . . . . . 3
G a t h e r i n g t h e Judgments . . . . . . . . . . . . . . . . . . . . 3
Analyzing the Data . . . . . . . . . . . . . . . . . . . . . . . 4
R e s u l t s o f t h e S t u d y . . . . . . . . . . . . . . . . . . . . . . . . 5
C h a r a c t e r i s t i c s o f t h e P a n e l i s t s . . . . . . . . . . . . . . . . 5
Judgments about Relevancy and Opportunity t o Learn . . . . . . . 5
Mathemati cs Test . . . . . . . . . . . . . . . . . . . . . . 5
ReadingTest . . . . . . . . . . . . . . . . . . . . . . . . 6
W r i t i n g M u l t i p l e Choice Test . . . . . . . . . . . . . . . . 6
Suinrnary f o r Mu1 ti ple Choice Tests . . . . . . . . . . . . . 7
Essay Question o f the W r i t i n g Test . . . . . . . . . . . . . 7
Judgments About Passing Scores . . . . . . . . . . . . . . . . . 7
M u l t i p l e C h o i c e Tests . . . . . . . . . . . . . . . . . . . 7
EssayQuestion . . . . . . . . . . . . . . . . . . . . . . . 8
Comments Related t o Possible Bias . . . . . . . . . . . . . . . 8
Conclusions and Recommendations . . . . . . . . . . . . . . . . . . 9
The V a l i d i t y of the PPST f o r Use i n Arizona . . . . . . . . . . 9
S e t t i n g a Standard o f Performance . . . . . . . . . . . . . . . 9
LIST OF TABLES
NumbUern iovfe rPsai tnye l .M.em.be.rs .fr.om. E.a.ch .E.mp.lo.yin.g .D.i s.t r.i c.t o.r . . . . . 12 C h a r a c t e r i s t i c s o f Panel Members . . . . . . . . . . . . . . . . 13 Math Test .R elevancy .E thnic Groups . . . . . . . . . . . . . 14
Math Test . Relevancy I Education Involvement Groups . . . . . . 15 Math Test .O pportunity .E thnic Groups . . . . . . . . . . . . 16 Math Test -- Opportunity .E ducation Involvement Groups . . . . . 17 Math Test .R elevancy and Opportunity .A l l Panelists . . . . 18
TABLE OF CONTENTS (CONTINUED)
Tab1 e -P- age--
Reading Test - Relevancy - Ethnic Groups . . . . . . . . . . . . 19 Reading Test -- Relevancy - Education Involvement Groups . . . . 20 Reading Test - Opportunity - Ethnic Groups . . . . . . . . . . . 21 Reading Test - Opportunity - Education Invol\/ement Groups . . . 22 Reading Test - Relevancy and Opportunity - A l l Panelists . . . . 23
W r i t i n g M u l t i p l e Choice Test - Relevancy - Ethnic Groups . . . . 24
N r i t i n g M u l t i p l e Choice Test - Relevancy - Education Involvement Groups . . . . . . . . . . . . . . . . . . . . . 25 W r i t i n g M u l t i p l e Choice Test - Opportunity - Ethnic Groups . . . 26
W r i t i n g M u l t i p l e Choice Test - Opportunity -- Education InvolvementGroups . . . . . . . . . . . . . . . . . . . . . 27
W r i t i n g M u l t i p l e Choice Test - Relevancy and Opportunity - A l l P a n e l i s t s . . . . . . . . . . . . . . . . . . . . . . . 28
A l l M u l t i p l e Choice Tests - Percent o f Items Viewed by
Specified Percent o f Judges r e Relevancy and Opportunity . . 29 Need and Opportunity t o Write - Percent o f Ethnic Groups . . . . 30
Need and Opportunity to Write - Percent o f Education InvolvementGroups . . . . . . . . . . . . . . . . . . . . . 31
A l l M u l t i p l e Choice Tests - Number of Judges Choosing D i f f e r e n t Passingscores . . . . . . . . . . . . . . . . . . . . . . . 32
A l l M u l t i p l e Choice Tests - Average Passing Score - Both
Groupings and A l l Panelists . . . . . . . . . . . . . . . . 33
Essay P o r t i on o f W r i ti ng Test - Average Pass i ng Score - Both
Groupings and A l l Panelists . . . . . . . . . . . . . . . . 34
Estimated F a i l u r e Rates f o r Three Sets o f Possible Passing
S c o r e s . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Possible Arizona Passing Scores and Passing Scores Used i n
Other Pl dces . . . . . . . . . . . . . . . . . . . . . . . . 36
APPENDICES
A Test Specifications f o r the NTE Pre-Professional S k i l l s Tests
B Letters Requesting Nominations o f Possible Panelists
C Population o f Arizona by Raci a1 /Ethnic Group
D Written I n s t r u c t i o n s and Response Sheets Used i n Gathering Data
E Outline o f the Oral I n s t r u c t i o n s Presented t o the Assembled
Panel i s t s
F Guidelines Used t o Score PPST Essays
-I N-T R-O-D I-J C-T I O N
BACKGROUND ---.-
I n 1984 the Arizona L e g i s l a t u r e enacted a b i l l r e q u i r i n g a l l applicants for
admission t o teacher education programs at the three s t a t e u n i v e r s i t i e s t o pas3
a basic s k i l l s t e s t i n mathematics, reading, and grammar. A f t e r considering
many a l t e r n a t i v e s , the Arizona Board o f Regents decided t o use the Pre-
Professional Ski 11 s Tests (PPST) developed by the Educational Testing Service
(ETS) as p a r t o f i t s National Teacher Examination programs t o s a t i s f y the statu-t
o r y requirement. Although these t e s t s have been validated i n and are c u r r e n t l y
used i n a number of other states (Delaware, Kansas, Nevada, Tennessee, Texas,
and West V i r g i n i a ) , it was necessary t o be sure t h a t each o f the three t e s t s i s
appropriate f o r use i n Arizona. Thus, a content v a l i d i t y and standard s e t t i n g
study was conducted i n May, 1985.
DESCRIPTION OF THE P-P--S-T
The Pre-Professional S k i l l s Tests assess basic p r o f i c i e n c i e s i n reading,
w r i ting, and mathematics. A1 together, t h e y i n c l u d e 125 mu1 t i p l e choice @ questions and the w r i t i n g of one short essay, r e q u i r i n g a t o t a l o f two and me-h
a l f hours f o r a l l portions o f the exam. Separate scores are reported f o r
reading, mathematics, and w r i t i n g . The performance on the w r i t t e n essay i s
included i n the w r i t i n g score.
The reading s k i l l s t e s t e d i n c l u d e l i t e r a l comprehension, the a b i l i t y t o
understand how material i s organized and how it conveys the message, as we1 1 as
the a b i l i t y t o make reasoned qua1 i t a t i v e judgments about the nature and merits
OF a w r i t t e n message. The mathematical competencies t e s t e d i n c l u d e knowledge
acquired from having studied mathematics from elementary through secondary
school. The w r i t i n g s k i l l s t e s t e d i n c l u d e an a b i l i t y t o use grammar and
language appropriately and t o communicate i n w r i t i n g w i t h a s p e c i f i c aim or pur-pose
i n mind.
The reading t e s t i s composed o f 40 m u l t i p l e choice items and requires 40
minutes. The mathematics t e s t i s composed o f 40 m u l t i p l e choice items r e q u i r i n g
50 minutes. The w r i t i n g t e s t includes 45 m u l t i p l e choice items r e q u i r i n g 30
minutes and the w r i t i n g o f one essay i n 30 minutes. A complete set o f t e s t
s p e c i f i c a t i o n s i d e n t i f y i n g the precise s k i l l s sampled i n each of t h e t h r e e t e s t s
i s contained i n Appendix A.
a OVERVIEW -OF THE STUDY
To a s c e r t a i n t h e content v a l i d i t y of the three t e s t s and t o qather i n f o r -
mation relevant t o the establ ishment- o f reasonable passing scores, a repre-sentative
panel of Arizona teachers and teacher educators was assembled t o make
c e r t a i n judgments about each item o f each t e s t .
For each of the m u l t i p l e choice questions, the judgments r e q u i r e d t h e p a ~ e l -
i s t s t o r a t e the extent t o which the knowledge or academic s k i l l tested was
relevant t o competent performance, t o i n d i c a t e whether the t y p i c a l applicant
would have had an opportunity t o acquire the knowledge or s k i l l required, and t o
i n d i c a t e the number out o f 100 m a r g i n a l l y q u a l i f i e d i n d i v i d u a l s who would be
able t o answer the question c o r r e c t l y . For the essay p o r t i o n o f the w r i t i n g
t e s t , the p a n e l i s t s were f i r s t asked whether a teacher i n Arizona needed t o be
able t o w r i t e an acceptable essay and whether applicants had an opportunity t o
learn t o w r i t e an acceptable essay. Then the p a n e l i s t s were asked t o read
twelve essays which had been w r i t t e n by i n d i v i d u a l s who had taken the t e s t i n
other states and t o select two which represented the minimum l e v e l of s k i l l t h a t
a new teacher or an applicant f o r a teacher education program i n Arizona should
demonstrate. A d d i t i o n a l l y , p a n e l i s t s were given an opportunity t o make w r i t t e n
co~nments about each item and were urged t o record s p e c i f i c information about
any item t h a t included language t h a t may r e s u l t i n a b i a s against an ethnic
m i n o r i t y .
An analysis of the responses o f the p a n e l i s t s as a whole and of subgroups of
the p a n e l i s t s based on e t h n i c i t y and type o f involvement i n education indicated
@ t h a t a large m a j o r i t y o f the items on each of the m u l t i p l e choice t e s t s were
relevant and t h a t applicants would have had an opportunity t o acquire the
knowledge or s k i l l required. Similar r e s u l t s were found for the essay questior~.
The analysis o f the judgments about the percent o f i n d i v i d u a l s who would get
t h e i t e m correct and o f the judgments about the essays selected as representing
the performance o f a m i n i m a l l y q u a l i f i e d i n d i v i d u a l l e d t o an estimation o f
passing scores appropriate f o r Arizona. Those scores were w i t h i n a few points
o f the scores used i n other s t a t e s .
F i n a l l y , a t a b u l a t i o n o f the f r e e response comments made about each i t e n
revealed t h a t very few questions had any patenti a1 f o r bias. Iq each case, it
wor~ld appear t h a t a minor change i n t h e i t e m could avoid the p o s s i b i l i t y o f a
problem. The two items t h a t received more than one comment are not included i n
any subsequent form of the PPST. The objectionable language i n the two i t e m
would not have led t o the s e l e c t i o n of an i n c o r r e c t answer.
P R O C E D U R E S F O L L - O W E D
OBTAINING THE REVIEW PANELISTS
I n s e l e c t i n g p a n e l i s t s t o p a r t i c i p a t e i n the study, care was taken t o ensiwo
that a l l Arizona constituencies were a p p r o p r i a t e l y represented. A l e t t e r
requesting nominations (see Appendix B) o f possible p a n e l i s t s was sent t o the
superintendent of every school d i s t r i c t i n Arizona and t o the president o f each
o f the three s t a t e u n i v e r s i t i e s i n Arizona. A request was made i n the l e t t e r
that the nominees be representative o f a l l ethnic groups and of both genders,
t h a t thc nominees represent a v a r i e t y o f grade l e v e l s and subject matter taught,
t h a t thee iiricf ude p r i n c i p a l s as well as classroom teachers, arld t h a t the nomi-nees
be s e n s i t i v e t o the needs o f students from various c u l t u r a l backgrounds.
The superintendent or president was asked t o i n d i c a t e the gender, e t h n i c i t y ,
grade l e v e l , and subject taught (or type o f p o s i t i o n ) o f each nominee.
The f i n a l determination o f which nominees t o i n v i t e t o p a r t i c i p a t e was rnadc
a f t e r c a r e f u l l y consid2ring the r e s u l t s of the 1980 census which described the
population o f Arizona by r a c i a l / e t h n i c group (see Appendix C ) . I n v i t a t i o n s were
sent t o nominees who would be representative o f ethnic groups, of genders, o f
school grade l e v e l and subject matter taught, and o f the l o c a t i o n o f employment
( r u r a l and urban i n the case o f teachers and p r i n c i p a l s and each o f the three
s t a t e u n i v e r s i t i e s i n the case o f teacher educators). Whenever the i n d i v i d u a l
o r i q i n a l l y selected could not accept the i n v i t a t i o n , a rep1 acemenl was "iond a whose c h a r a c t e r i s t i c s matched the person o r i g i n a l l y i n v i t e d as c l o s e l y as
possible.
GATHERING T---H----E-- JUDGMENTS
The p a n e l i s t s were assembled at Arizona State U n i v e r s i t y on A p r i l 25, 1985.
A f t e r the p a n e l i s t s were given a b r i e f background f o r the study, D r . Richard E.
Peterson, Senior Research Psychologist from the Berkeley F i e l d Service O f f i c e o f
the Educational Testing Service, described the development o f the t e s t s i n
general and reviewed the procedures followed by ETS t o ensure t h a t the t e s t
items were f r e e o f c u l t u r a l bias.
Following t h e o r i e n t a t i o n , the s p e c i f i c r a t i n g scales and response sheets t o
be used were reviewed w i t h the panelists. They were encouraged t o ask questions
u n t i l they were clear about the required tasks. The p a n e l i s t s Mere next asked
t o practice the task by making appropriate judgments about a reading t e s t i tern
used as a sample question i n the 1984-1985 B u l l e t i n o f Information about the
PPST published by ETS.
GATHERIN.G- - THE JUDGMENTS( CONTINUED)
To be sure that everyone understood the basis for the judgments, several
panelists were asked t o reveal their judgments about the sample item and t~
indicate what led them t o their conclusion. The ensuing discussion centered
primarily around the question of whether the judgments about the proportion o f
individuals who would get the item correct should be based on what would happen
or on what should happen. The validation study consultant indicated that a
panel i st ' s judgment about what an idea1 i zed conceptual group of rnargi nal ly
qualified individuals would do and a panelist? judgment about what a group of
marginally qualified -r-e al individuals should do amounted to the sarne thing. The
panelists then proceeded to make their own individual judgments about each
multiple choice item.
In the afternoon, the panelists received instructions for reading and
judging the twelve essays. Following a short question and answer period t o be
certain that everyone understood the judgments which were to be made about the
essays, the panel ists were asked to finish their judgments about the multiple
choice items if they had not already done so and then to proceed to complete
their work with the essays. Each panelist was allowed t o leave after completing
all tasks and returning all materials received.
A copy of all written instructions and response sheets used i n gathering
data is in Appendix D. An outline of the oral instructions presented t o the
panelists is in Appendix E.
-A--N--A---L YZING THE DATA
The relevancy of each item of each multiple choice test was identified by
calculating the percent of the members of each ethnic subgroup who judged the
item as at least slightly relevant. Similar percentages were calculated for
each group of panelists categorized according to type of involvement in educa-t
i on (elementary teachers and principals, secondary teachers and principals, qr
teacher educators). Additionally, the percent of the total group who judged
each item as very relevant, as very or moderately relevant, or as very, moder-ately,
or slightly relevant was also determined. To ascertain the content
validity of each multiple choice test, the percent of items on each test that
was viewed by 50 percent or more, by 67 percent or more, and by 75 percent or
more of the panelists as falling in each of the relevancy categories described
above was computed.
Similar percentages were calculated for each of the subgroups and for the
total group i n order to answer the question as t o whether applican'ts would have
had an opportunity to acquire the knowledge or ski 11 required by each multiple
choice item and by each test as a whole. Similar percentages were also calcu-lated
to reveal the proportion of respondents from the various subgroups and for
the total group who said that teachers did need to be able to write an essay and
that applicants would have had an opportunity to learn to write an essay. This
ANALYZING-. THE DATA (CONTINUED)
@ information was used t o assess the v a l i d i t y o f the essay section o f the w r i t i n g
t e s t . Every f r e e response comment was examined, w i t h p a r t i c u l a r emphasis on
those responses which were relevant t o a possible bias.
Estimates of a reasonable passing score were obtained f o r each m u l t i p l e
choice t e s t by the Tucker/Angoff method. This method involves c a l c u l a t i n g a
passing score f o r each p a n e l i s t based on h i s o r her judgment of each item. The
passing score recommended by the panel as a whole i s obtained by averaging the
passing scores calculated f o r each p a n e l i s t . The complete frequency d i s t r i b d -
t i o n of the judges' passing scores f o r each t e s t was tabulated and exanined t o
be sure t h a t no p a n e l i s t misunderstood the d i r e c t i o n s or was making extrene and
unwarranted judgments. Estimates o f a reasonable passing score were s i m i l a r l y
computed f o r each o f the subgroups based on e t h n i c i t y and on involvement i n
education.
The score on the essay p o r t i o n o f the w r i t i n g t e s t i s the sum o f the r a t i n g s
o f two t r a i n e d essay evaluators. The r a t i n g s are on a s i x - p o i n t scale. There-f
o r e , t h e sum o f the r a t i n g s previously received by the two essays selected by
the p a n e l i s t as r e p r e s e n t i n g t h e minimum l e v e l o f w r i t i n g s k i l l a new teacher or
an applicant f o r a teacher education program i n Arizona should have was used as
t h a t panel i s t ' s es tirnated standard o f performance. The average passing score
was then computed over a11 reviewers and f o r each of the subgroups o f p a n e l i s t s
previously described.
-R -E S U L--T S O F T H E --S--T U D Y
CHARACTERISTICS OF THE PANELISTS
A t o t a l o f 56 p a n e l i s t s p a r t i c i p a t e d i n the review process. A complete l i s t
o f the employing i n s t i t u t i o n s and the number o f p a n e l i s t s from each i s presented
i n Table 1. The number and percent o f p a n e l i s t s from various subgroups can be
found i n Table 2. It i s apparent from these two tables t h a t the review pane?
members adequately represented the major Arizona constituencies.
JUDGMENTS ABOUT RELEVANCY AND OPPORTUNITY TO LEARN --. ---- -.-
-Mat- hem at ic s Test
The percent o f the members o f d i f f e r e n t ethnic groups who judged each item
o f the mathematics t e s t as at l e a s t s l i g h t l y relevant i s presented i n Table 3.
The same information f o r groups c l a s s i f i e d according t o the type of involvement
i n education i s given i n Table 4. An examination of these two tables i n d i c a t e s
t h a t a s o l i d m a j o r i t y o f every subgroup viewed each item as relevant. T t i s
also apparent t h a t i n the vast m a j o r i t y o f the cases, the proportion who judged
the item as relevant was at or near 100. The averaqe at the bottom of these
t a b l e s i n d i c a t e s t h e p r o p o r t i o n o f items i n the t e s t judged as a t l e a s t s l i g h t l y
relevant by t h e i n d i c a t e d subgroups.
JUDGMENTS ABOUT RELEVANCY AND OP-P-O-R-T UNITY TO LEARN -(C -O- NTINUED)
Mathematics Test (-C-on-tin ued)
Similar information about t h e o p p o r t u n i t y t o acquire the knowledge or s k i l l
required by the item i s presented i n Tables 5 and 6. With two exceptions, 50
percent or more o f every subgroup concluded t h a t applicants would have had an
opportunity t o acquire the knowledge or s k i l l necessary t o get the item c o r r e c t .
Tile two excep'tions are f o r item 32 where only 43 percent of the Black panel-i
s t s f e l t that there would have been an opportunity t o learn and item 26 where
neither of the two members of the "other ethnic" group f e l t there would have
been an opportunity t o acquire the necessary information before t a k i n g t h e t e s t .
More s p e c i f i c in-formation w i t h respect t o relevancy and information about
opportunity t o learn i s presented f o r a l l p a n e l i s t s combined i n Table 7. In
every case, 79 percent or more o f a1 l panel is.ts saw the item as relevant and 63
percent or more o f a l l p a n e l i s t s f e l t the applicants would have had an oppor-t
u n i t y t o acquire the knowledge or s k i l l required by the item.
Reading Test
p-
Information i d e n t i c a l t o t h a t j u s t described f o r the mathematics t e s t i s
presented f o r the reading t,est i n Tables 8 through 12.
Careful examination of these tables reveals an even greater relevancy and
opportunity t o learn f o r the reading t e s t than was found f o r the mathematics
t e s t . The only instances i n which the percentage was 50 or smaller f o r any
subgroup occurred i n the subgroup composed of only two members. With a l l groups
combined, the percentage t h a t judged the item as relevant was 84 or larger f o r
every item. The percentage of p a n e l i s t s who i n d i c a t e d t h e r e would have been an
opportunity t o acquire the knowledge or s k i l l required was 68 or larger i n every
case.
W r i t i n g Mu1 ti p l e Choice Test
Informat-ion about the m u l t i p l e choice p o r t i o n o f the w r i t i n g t e s t s i r n i l a r t o
t h a t presented f o r the mathematics and the reading t e s t s can be found i n
Tables 13 through 17. Once again, it i s clear t h a t a vast m a j o r i t y o f the iteins
are seen as relevant by every subgroup and t h a t when the r e s u l t s f o r a l l
p a n e l i s t s are combined, t h e i t e m i s seen as relevant by at l e a s t 82 percent of
the group i n every case.
Summar- for MultipleChoice Tests -.- - P
A summary of the results with respect to relevancy and opportunity t o
acquire the knowledge or skill required is presented in Table 18 for all tiree
multiple choice tests. I t is clear that 68 percent or more of the items on
every test were individually judged by 75 percent or more of the panelists as
very relevant or moderately relevant, and every item on every test was seen by
75 percent or more of the panelists as at least slightly relevant. I t can also
be seen that every item on every test was seen by at least a majority of the
panelists as requiring knowledge or skills which applicants would have had a
chance to learn. Indeed, no fewer than 87 percent of the items on any test were
seen this way by 75 percent or more of the reviewers.
Information about the relevancy of being able to write an essay and about
the opportunity to learn to write is presented in Tables 19 and 20. From these
tables, i t can be seen that at least 82 percent of every ethnic subgroup believes
that teachers or students in a teacher education program need to be able to
write an acceptable essay and that at least 74 percent of every subgroup based
on involvement in education sees i t in the same way. For all panelists com-bined,
85 percent see the skill of writing an essay as relevant. With respect
t o the opportunity to learn to write an essay, no fewer than 76 percent of any
subgroup and 84 percent of all panelists believe that applicants would have had
an opportunity to acquire the skill.
JUDGMENTS ABOUT PASSING SCORE-S
Multiple Choice Tests
The complete frequency distribution of the estimated passing scores obtained
by the TuckerIAngoff method for each panelist is presented for each of the three
multiple choice tests in Table 21. Although there were a few extreme judgments
in the case of the writing t e s t , the overall shape of the distributions was such
that i t seemed most reasonable t o use the arithmetical means of the results froin
the different panelists to arrive at an overall, estimated, reasonable passing
score. The results of averaging the values for all panelists are presented in
raw score form for each test and for subgroups as well as for all reviewers com-bined
i n Table 22.
Essay Que-. stion
A reasonable passing score was estimated from the c o l l e c t i v e responses o f
the panelists. The average o f these estimates f o r d i f f e r e n t subgroups and f o r
a l l p a n e l i s t s combined i s given i n Table 23. It i s apparent t h a t no subgroup
passing score deviated from the average f o r a l l p a n e l i s t s by more than one
p o i n t . The average o v e r a l l passing score o f nine, which represents the sum o f
the r a t i n g s o f two t r a i n e d readers, would set as a reasonable standard an essay
which, according t o the scoring guidelines, would f a l l h a l f way between one
described as " c l e a r l y demonstrates competence i n w r i t i n g , though i t may have
m i nor e r r o r s " and one described as "demonstrates competence i n w r i - t i ng, though
it qay have occasional errors". A complete scoring guide describing a l l l e v e l s
o f performance on the essay question i s found i n Appendix F.
-C- OMMENTS RELATED TO POSSIBLE BIAS
Although many comments were made about the items on the three m u l t i p l e
choice t e s t s reviewed, only 28 comments suggested a possible problev w i t h bi3s.
These 28 comments were made by only 13 d i f f e r e n t p a n e l i s t s ( i n c l u d i n g a t l e a s t
one p a n e l i s t frorn each ethnic group) and r e f e r r e d t o 25 d i f f e r e n t itpms. Only
two items received more than one comment. Four comments were made about: itclrn 34
on the w r i t i n g t e s t and three comments were made about i terri 13 on the reading
@ t e s t . (ETS has reported t h a t both o f these t e s t items have been removed from a l l forms of the t e s t s c u r r e n t l y i n use. I n addition, there was one comment
about the mathematics t e s t i n general, two comments about the reading t e s t i n
general, and four general comments about the w r i t i n g m u l t i p l e choice t e s t . Yo
comments were made about the essay question.
Two o f the comments about item 13 on the reading t e s t concerned the def i n i -
t i o n o f the word, "poncho". The t h i r d comment pointed out t h a t some Native
Americans may never have seen a crossing guard. A review o f item 13 revealed
t h a t a student did not need t o know the d e f i n i t i o n o f "crossing guard" or o f
"poncho" t o answer the question c o r r e c t l y . The general comments about the
reading t e s t indicated t h a t c e r t a i n word choices and the wording o f long
paragraphs might pose some d i f f i c u l t i e s for l i m i t e d English speakers.
A1 1 f a u r of the comments about item 34 on the w r i t i n g t e s t r e f e r r e d t o the
unneces5ary use o f the personal pronoun '%she" i n one o f the a1 t e r n a t i v e
responses. The four general comments made about the w r i t i n g t e s t r e f e r r e d again
t o the p o s s i b i l i t y t h a t the word choice and sentence s t r u c t u r e of some of the
items might make the items d i f f i c u l t f o r non-native speakers of English.
The one comment made about the mathematics t e s t i n general indicated t h a t
Inany o f the ski1 1s may not be retained by a Native American and t h a t Native
American students would not complete the t e s t unless they had had three or- more
years o f mathematics i n high school.
C O N C L U S -I- O N S A N D R E C O-M M E N D A T I O N S
THE VALIDI-T--Y-- OF THE PPST FOR USE I N ARIZONA
There i s considerable evidence t h a t a s u b s t a n t i a l m a j o r i t y o f a l l p a n e l i s t s
and o f most of the relevant subgroupings o f p a n e l i s t s judged the i n d i v i d u a l items
on a l l the t e s t s as relevant t o the performance o f students i n educational
programs and o f teachers i n Arizona. Thus, it can be said t h a t the PPST has
considerable content v a l i d i t y f o r use as an entrance requirzment t o any o f the
three s t a t e u n i v e r s i t i e s . It i s u n l i k e l y t h a t any other c u r r e n t l y a v a i l a b l e
t e s t would show greater content v a l i d i t y than the PPST.
Another aspect o f the v a l i d i t y i s whether the t e s t i s v a l i d f o r the d i f f e r -
ent groups w i t h whom it i s l i k e l y t o be used. Once again, the evidence pre-sented
here i n d i c a t e s t h a t each o f the three t e s t s has considerable v a l i d i t y f o r
the subgroups examined i n t h i s study. The conclusion t h a t the t e s t s are v a l i d
For use w i t h groups who are l i k e l y t o have d i f f e r e n t involvement i n education
stems from the f a c t t h a t elementary teachers and p r i n c i p a l s , secondary teachers
and p r i n c i p a l s , and teacher educators a l l see the items as relevant t o success-f
u l performance. On the other hand, evidence t h a t the t e s t s are v a l i d f o r use
w i t h d i f f e r e n t ethnic groups must depend upon evidence t h a t the t e s t s contain as
l i t t l e bias as possible, as well as evidence t h a t the items are relevant t o the
task f o r members of the d i f f e r e n t ethnic groups. Since no p o t e n t i a l bias was
i d e n t i f i e d by any p a n e l i s t i n 80 percent o f the items and only two items were
c a l l e d i n t o question by more than one reviewer, it would seem reasonable t o
@ conclude t h a t the t e s t s are l i k e l y t o be as f r e e from bias as current t e s t
construction technology would allow.
Although not d i r e c t l y r e l a t e d t o the question o f the content v a l i d i t y o f the
t e s t s , an issue of l e g a l and s o c i a l concern i s whether or not those who w i l l be
t a k i n g the t e s t s would have had an opportunity t o acquire the necessary knowl-edge
or s k i l l before they are required t o pass the examination. The evidence
presented here i n d i c a t e s t h a t most p a n e l i s t s b e l i e v e t h a t the t y p i c a l applicant
would have had an opportunity t o l e a r n t h e ski1 1s necessary t o answer most of
the items c o r r e c t l y . However, t o the extent t h a t some p a r t i c u l a r i n d i v i d u a l s
may not have had such an opportunity, it would seem reasonable t h a t some form o.f
a d d i t i o n a l t r a i n i n g should be a v a i l a b l e t o those who f a i l one or more o f the
t e s t s on the f i r s t t r y .
SETTING A PASSING SCORE
Whenever any t e s t i s used f o r c e r t i f i c a t i o n or f o r admission t o a program, a
passing score must be specified. Because a passing score represents a value
judgment as t o how much it i s worth t o be at a c e r t a i n p o i n t on a scale, there
i s no mathematical way t o s p e c i f y such a p o i n t . If the standard i s set too low,
s o c i e t y i s not protected because some u n q u a l i f i e d i n d i v i d u a l s may be allowed t o
teach; if the standard i s set too high, then some i n d i v i d u a l s who might have
the necessary basic s k i l l s w i l l not be permitted t o teach. Only a human being
can judge the r e l a t i v e seriousness of these two types o f errors.
One approach t o e s t a b l i s h i n g a reasonable passing score i s to have a repre-sentative
group o f i n d i v i d u a l s f a m i l i a r w i t h the task i n d i c a t e t h e l e v e l of per-formance
t h a t they t h i n k a minimally q u a l i f i e d p r a c t i t i o n e r would be able t o
achieve. Since, however, no two i n d i v i d u a l s are l i k e l y t o make exactly the same
judgment, it i s necessary t o aggregate the judgments i n the form o f an average.
That procedure was followed i n t h i s study.
The passing scores reported i n Tables 22 and 23 represent estimated stan-dards
i n the form of raw true scores. Before they can be used, they must be
t r a n s l a t e d i n t o scaled scores which are the ones reported t o applicants and t o
the agencies u s i n g t h e scores by ETS. This t r a n s l a t i o n i s accomplished by means
o f NTE Score Conversion Tables PW 4/11/84 provided by ETS. The t r a n s l a t i o n
r e s u l t s i n o n l y t h r e e scores since the w r i t i n g m u l t i p l e choice items and the
w r i t i n g essay item are combined i n t o a s i n g l e composite. The raw composite
score i s obtained by adding the number o f correct answers on the m u l t i p l e choice
section (maximum possible score: 45) t o 3.75 times the score on the essay
(maximum essay score: 12). This r e s u l t s i n approximately 60 percent weight i n y
for the m u l t i p l e choice section and approximately 40 percent weighting f o r the
essay section. The converted scores f o r each o f the three t e s t s are reported i n
the f i r s t row o f Table 24.
I n s e t t i n g passing scores, it i s generally considered appropriate t o make
one a d d i t i o n a l adjustment. This adjustment takes i n t o consideration the e r r o r
o f measurement inherent i n any psychological o r educational t e s t . To be sure
t h a t candidates are not excluded because o f an e r r o r of measurement when t h e i r
t r u e performance i s j u s t at the lowest acceptable level, it i s common p r a c t i c e
t o set the actual passing scores one or two standard e r r o r u n i t s below t h a t
judged as appropriate by the reviewing panel. The standard e r r o r of the PPST i s
approximately t h r e e p o i n t s on the scaled score.
Table 24 includes information about the estimated f a i l u r e rates f o r each of
the three t e s t s f o r each o f t h r e e p o s s i b l e sets o f passing scores. The f i r s t
set of passing scores represents the standards of performance judged t o be rnini-ma1
by t h e p a n e l i s t s . The second set o f passing scores represents those which
f a l l three points (approximately one standard e r r o r u n i t ) below the minimal
l e v e l , and the t h i r d set represents those which f a l l s i x p o i n t s (approximately
two standard e r r o r u n i t s ) below the estimated minimal performance level.
Two estimates of the f a i l u r e rates are provided. One estimate i s based on
the administration o f the PPST t o over 8,000 examinees from February 1983 t o
February 1984. The second estimate i s based on the r e s u l t s of over 4,000 exam-inees
who took the t e s t s i n the summer o f 1984.
e SETTING A PASSING SCORE (CONTINUED) ---Be-ca-us-e -i t is also helpful to see how the three possible sets of passing
scores compare with those used in other places, Table 25 has been prepared. T Q
this table, the possible passing scores for Arizona are listed along with those
currently in use in Texas, in Delaware, in Tennessee, and at the University of
Cincinatti.
The final consideration when setting passing scores is whether a single corn-posite
score should be used or whether separate passing scores should be used
for each of the three tests. Whenever a single passing score is used, candi-dates
who are deficient in one area can compensate for this by doing excep-tionally
well in one or both of the other areas. Unfortunately, this means that
some people will be admitted who Fall far below the minimum standard for
accepted performance in one of the areas. If every area is judged by the
panelists to be relevant to successful perfor-mance, such a situation is not
tolerable. Therefore, separate passing scores for each of the three tests is
recommended. This recommendation is consistent with the requirements of
ARS 15-533.
TABLE 1. Number of Panel Members from Each
Employing District or University
Emp 1 oyer
Rural and Sma-ll Town Districts p-
Aqua Fria Union High School District #216
Bull head City Elementary District #l5
Bureau of Indian Affairs School
Chino Val 1 ey Elementary District #51
Dill, Mary E. Elementary District #41
Dysart Unified District #89
Globe Unified District #1
Marana Unified District #6
Morenci Unified District #l8
Nogales Unified District #I
Pearce Elementary District #22
Quartzsite Elementary District #4
Round Valley Unified District #10
Santa Cruz Valley Union High School District #840
Tuba City Unified District #15
Vail Elementary District #20
Window Rock Unified District #8
Winslow Unified District #1
TOTAL, Rural Districts
Urban and Suburban Districts
Amphitheater Unified District #LO
Deer Valley Unified District #97
Flagstaff Unified District #1
Flowing Wells Unified District #8
Glendale Elementary District #40
Kyrene Elementary District #28
Litchfield Elementary District #79
Mesa Unified District #4
Paradise Valley Unified District #69
Pendergast Elementary District #92
Phoenix Union High School District #21O
Scottsdale Unified District #48
Sunnyside Unified District #12
Tanque Verde Unified District #I3
Tucson Unified District #1
Washington Elementary District #6
TOTAL, Urban Districts
-Univ-ers-iti es
Arizona State University
Northern Arizona University
University of Arizona
Number o f
-P--a- nel Members
TOTAL, Universities
TABLE 2. Character i s t i cs of Panel Meinhers
Characteristic -. -
Gender:
Femal c
Ma1 e
Ethnic Group:
American Indian
Black
G aucas i an
Hispa~ic, Mexican American
Other
Type of Educator:
Elementary Teacher
Elementary Principal
Other Elementary Personnel
TOTAL, Elementary
Secondary Teacher
Secondary Principal
Other Secondary Personnel
TOTAL, Secondary
TOTAL, Teacher Educator
Subj cct Matter Area*
Science and Mathematics
English and Social Science
Other/Not Known/Several
Number -Perc-ent
a *For Secondary Teachers only.
I tem
Number
1
2
3
4
TABLE 3. Percent of Memhers o f Different Ethnic Groups
Who Saw the Item as at Least Slightly Relevant
for the Mathematics Test
Average
h e r i can
-In dian
Group
51 ack -Cauc-a -s ia- -n .- Hispa-ni-c
100
100
100
100
100
100
9 1
100
100
100
100
9 1
100
100
100
100
100
'3 1
100
100
9 1
100
100
100
91
91
100
9 1
100
100
8 2
91
7 3
9 1
100
7 3
9 1
9 1
8 2
9 1
Other
Ethnic
100
100
100
100
100
100
100
100
100
100
1Q0
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
I t em
-Nu-mb er
1
2
3
4
5
6
7
8
9
10
11
12
13
14
TABLE 4. Percent of Members with Differing Involvement
in Education Who Saw the Item as at Least
Slightly Relevant for the Mathematics Test
Average
Elementary
Teachers &
-Pr-in ci~--a-l s
loo
100
100
100
95
100
100
100
95
100
100
86
100
100
100
100
100
9 5
100
100
100
100
100
loo
90
100
9 5
90
90
95
86
86
86
95
100
8 6
90
90
90
95
Group
Secondary
Teachers P1
Principal-s
94
100
95
100
100
84
9 5
89
89
100
95
79
100
100
95
9 5
9 5
68
95
100
95
95
9 5
95
84
84
89
74
84
100
74
84
9 5
89
100
68
9 5
100
7 9
100
A1 1
Teachers &
Princii-a. - l s.-
97
100
97
100
97
93
97
95
92
100
97
8 2
100
100
97
9 4
97
82
97
100
97
98
97
97
8 7
92
9 2
82
88
97
80
8 5
9 0
92
100
7 7
92
95
85
97
Teacher
Educators ----
100
100
100
100
100
100
100
100
100
100
8 7
94
100
100
94
9 4
100
100
9 4
100
9 4
100
100
100
87
87
100
8 7
100
100
9 4
94
81
8 7
100
8 1
9 4
87
9 4
8 7
TABLE 5. Percent of Different Ethnic Groups Who Felt That
There Had Been an Opportunity to Learn the S k i l l
Required by the Mathematics Test Item
I tern
-N--t-~-r nbet-
Arner i can
-- Indian -
100
100
80
80
100
60
80
100
100
100
80
60
80
80
80
80
100
40
80
80
80
80
80
80
80
40
60
60
6 0
80
80
60
80
60
80
40
80
60
60
60
-- Group -
Black
100
100
100
100
100
8 6
86
100
7 1
100
100
71
7 1
100
86
100
100
7 1
86
100
57
100
86
100
86
86
7 1
57
86
7 1
7 1
4 3
57
86
100
5 7
86
100
7 1
86
Hisp-a nic-
100
100
100
100
100
100
9 1
100
73
100
100
9 1
100
100
100
9 1
100
8 2
100
100
91
9 1
100
100
9 1
64
9 1
73
82
9 1
91
7 3
91
100
100
55
9 1
9 1
6 4
64
Otbier
Ethnic
100
100
100
100
100
5 0
100
100
50
100
100
100
100
100
100
100
100
50
100
100
100
100
100
100
100
0
5 0
5 0
100
100
100
100
5 0
100
100
5 0
100
100
5 0
100
Average
TABLE 6. Percent of Members with Differing Involvement
in Education Who Felt That There Had Been an
Opportunity t o Learn the Sk-i 1 l Required by the
Mat hernat i cs Pest I tern
I tern
-- Number
1
2
3
4
5
6
7
8
9
10
11
12
13
40
Average
Elementary
Teachers &
-P-r i tic im.,-
9 5
100
100
100
9 5
100
100
100
81
100
95
95
9 5
95
200
100
100
90
100
100
9 5
95
100
100
100
86
86
90
8 1
90
90
86
86
95
100
7 6
90
56
7 1
8 1
Group
Secondary
Teachers &
Principals-
A1 1
Teachers S(
-P-r -i nc- 1 2als -
9 4
100
100
100
9 8
90
9 3
100
8 5
9 7
95
80
95
94
9 5
98
98
80
9 5
98
5 3
9 3
9 3
9 8
90
78
80
8 3
8 5
90
9 3
80
80
93
98
68
8 5
93
70
80
89.2
Teacher
Educators
100
100
9 4
100
100
69
100
94
88
100
88
9 4
100
100
9 4
9 4
100
75
9 4
100
9 4
88
9 4
3 3
9 4
7 5
75
6 3
8 1
9 4
88
8 1
69
7 5
100
5 0
9 4
88
75
69
88.1
I' te~n
Number
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Average
% Very
Re1 evant
79
86
70
88
75
3 8
5 2
46
43
61
6 3
3 9
68
75
5 9
61
55
2 0
54
7 1
52
43
45
3 9
38
3 2
34
14
23
46
29
29
2 1
3 3
5 4
14
20
55
7
30
Judgnlents of Panel Members w i t h Respect
t o Relevancy and Opportunity t o Learn
f o r Each Item on the Mathematics Test
Re1 evancy
% Very or
Moderately % Very, Moderately,
Re1 evant or Slightly Relevant
Opportunity
t o Learn
% Ye-s
98
100
98
98
130
84
95
9 8
86
98
93
8 4
9 5
96
93
96
9 8
7 9
93
9 8
88
91
9 1
96
8 9
7 5
7 7
7 5
8 6
9 1
9 1
80
73
8 8
9 8
6 3
88
9 1
7 1
7 7
I tern
Number
1
2
3
4
5
6
7
8
cj
10
I1
12
1.3
14
15
16
17
18
19
2 0
2 1
2 2
23
2 4
2 5
26
2 7
28
2 9
30
3 1
3 2
33
3 4
35
36
37
3 8
39
4 0
TABLE 8. Percent of Members of Different Ethnic Groups
Who Saw the Test Item as at Least Slightly
Relevant for the Reading Test
h e r i can
--I-n dian
100
100
100
100
80
100
80
80
80
100
60
80
100
80
100
80
100
100
100
100
80
80
100
100
80
100
100
100
80
80
100
100
80
100
100
100
80
80
80
80
-B---l-a ck
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
86
100
1.00
86
100
86
8 6
100
100
loo
100
100
100
86
100
86
86
180
100
100
86
100
86
100
100
-Cau-cas ian Hispanic
-
Other
-Et-h nic
100
100
10!3
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
50
100
100
100
100
100
100
100
100
100
50
100
100
100
100
100
5 0
50
100
Average
TABLE 9. Percent of Members with Differing Involvement
in Education Who Saw the Itern as at Least
Slightly Relevant for the Reading Test
I tom
-N--u-r nber
1
2
3
4
5
6
7
8
9
10
I1
12
13
14
15
16
17
18
13
2 0
2 1
22
23
24
25
26
2 7
28
2 9
30
31
3 2
33
34
3 5
36
37
3 8
39
40
Average
Elementary
Teachers P1
Pri ncda-l s
100
100
100
100
100
100
100
100
100
100
95
100
100
100
100
9 5
100
9 5
9 5
100
100
9 5
100
100
100
100
100
100
9 5
100
100
100
100
100
9 5
100
100
100
100
100
Group
Secondary
Teachers &
Princip-a-ls
----------
A1 1
Teachers &
-Prin-cip -als
Teacher
Ed~~c-a tors
100
100
100
100
94
100
94
9 4
9 4
9 4
100
87
87
9 4
8 7
'3 4
9 4
9 4
7 5
94
9 4
87
37
100
8 7
9 4
92
100
7 5
100
100
6 9
9 4
9 4
87
100
94
8 7
8 7
96
TABLE 10. Percent of Different Ethnic Groups Who Felt That
There Had Been an Opportunity to Learn the Ski1 1
Required by the Reading Test Itern
I tern
Nunihcr
Aver age
American
Indian -
80
80
100
100
80
80
60
80
40
80
100
80
80
80
40
40
100
100
80
100
60
80
80
60
60
100
80
100
100
80
80
80
80
100
80
100
60
60
80
80
-B-la ck
100
100
86
100
100
100
71
8 1
86
7 1
86
86
7 1
7 1
7 1
71
100
100
7 1
100
100
86
57
100
100
86
100
100
86
86
86
86
7 1
7 1
7 1
loo
86
86
86
100
Group ---
Hispanic
100
100
100
100
9 1
91
100
100
82
73
100
82
9 1
100
9 1
100
100
100
8 2
91
9 1
100
73
73
73
9 1
9 1
91
73
9 1
91
64
6 4
9 1
82
100
9 1
91
100
100
Other
Ethnic
100
100
100
100
100
100
100
100
100
100
100
5 0
100
100
100
100
5 0
100
5 0
104)
5 0
0
0
100
100
5 0
5 0
5 0
100
5 0
100
0
5 0
50
0
100
5 O
0
5 0
5 0
TABLE 11. Percent of Members with Differing Iwolvement
i n Education Who Felt That There Had Reen an
Opportunity to Learn the Ski 11 Required by the
Read i ng Test I tem
:[ t e:n
Number
40
Average
Elementary
Tcaciiers &
-P-r incip-a -ls
100
100
100
9 5
90
9 5
9 5
95
90
90
9 5
100
90
95
90
90
100
95
90
100
90
86
90
30
95
100
100
95
95
9 5
9 5
86
90
90
86
9 5
86
86
100
100
Group
Secondary
Teachers &
Princi>al s
89
100
100
89
100
95
84
79
84
68
90
84
8 9
89
84
7 4
100
100
7 4
95
79
79
68
79
84
84
89
100
84
95
84
7 4
79
89
74
100
7 9
89
89
95
86.5
A1 1
Teachers &
-P-ri-u tci2a-l s
95
100
100
9 3
95
9 5
90
88
88
80
9 3
93
90
9 3
88
8 3
100
98
8 3
98
85
83
80
8 5
90
93
95
98
90
9 5
90
80
85
90
80
98
83
88
95
98
90.6
Teacher
-Edir-- c- a- to - r-s -
94
100
100
9 4
100
100
88
94
81
69
9 4
7 5
81
88
81
9 4
88
9 4
56
9 4
7 5
8 1
69
9 4
7 5
69
88
88
56
88
9 4
69
63
88
6 9
100
88
63
7 5
88
I t em
-Num-be r
. - @ Average
TABLE 12. Judgments of All Panel Members Combined
with Respect to Relevancy and Opportunity
to Learn for Each Item on the Reading Test
% Very
Re1 evant
75
7 5
71
80
70
68
50
68
5 3
44
55
4 6
61
7 1
69
67
80
82
3 6
74
5 2
3 9
43
64
6 3
58
7 1
64
3 6
69
48
36
46
4 1
3 2
50
5 3
47
6 9
7 6
----- Relevancy ----
% Very or
Moderately 7: Very, Moderately,
---R-- elevant or Slightly Relev-an-t
94 98
98 100
9 6 100
96 100
9 1 9 8
9 6 100
85 98
9 3 98
9 1 98
80 9 6
85 9 6
84 95
84 95
9 1 96
96 9 8
94 96
98 98
98 98
6 6 8 7
9 4 98
82 9 6
7 1 8 7
79 9 3
89 100
88 9 5
87 98
89 98
89 100
7 0 88
87 98
80 9 6
7 0 84
73 9 6
7 7 98
69 89
84 9 8
85 98
88 9 5
87 96
94 9 8
Opportunity
to Learn
----- % Yes
93
9 8
98
9 3
95
96
88
88
88
8 5
93
88
8 13
9 1
88
8 8
96
9 8
7 1
9 6
83
84
7 7
88
88
6 8
93
9 6
80
9 3
9 1
7 1
7 9
89
7 9
98
84
82
9 1
9 6
TABLE 13. Percent of Members o f Different Ethnic Groups
Who Savt the Item as at I-cast S l i g h t l y R ~ l ~ v a n t
for the dritiiig Multiple Choice lest
1 tern
-Num-be r
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2 0
2 1
2 2
23
2 4
2 5
26
2 7
23
2 9
3 0
3 1
3 2
33
3 4
3 5
36
37
38
39
40
41
42
43
44
45
Aver age
Amer i can
-I-n-d-i a n
100
100
100
100
100
100
100
100
100
100
100
80
80
100
80
80
80
80
100
80
80
60
80
60
100
100
100
100
80
80
100
100
100
100
80
80
100
100
100
100
100
100
100
100
100
92.4
B 1 aclc Caucasi arl
w----
100 9 7
100 94
100 9 7
100 8 7
100 100
100 100
100 100
100 94
100 97
100 9 7
100 97
100 100
100 97
100 9 7
100 100
100 100
100 97
100 100
100 100
100 94
86 9 7
100 100
100 94
86 90
86 81
86 100
100 90
100 100
100 100
100 100
100 100
LOO 100
100 100
100 9 7
100 94
100 94
86 90
86 84
86 90
100 94
100 94
100 97
100 90
100 9 7
100 90
97.8 95.7
-His-pan ic
9 1
100
100
100
91
100
100
8 2
100
91
100
100
91
91
100
100
100
100
9 1
91
91
100
82
64
91
100
9 1
100
100
9 1
100
100
100
91
100
100
90
91
100
100
100
100
100
9 1
91
95.2
Other
Ethn--i c
100
100
100
100
100
100
100
100
100
5 0
100
50
100
100
50
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
96.7
TABLE 14. Percent o f Members w i t h D i f f e r i n g Involvement i n
Education Who Saw the Itern as a t Least S l i g h t l y
Relevant f o r the W r i t i n g M u l t i p l e Choice Test
I .t em
Number
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Aver age
Elementary
Teachers &
P r i n c i p a l 5
9 5
100
100
95
100
100
100
95
100
100
100
100
100
100
100
100
100
100
100
I00
100
9 5
100
95
90
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
9 5
99.1
Group
Secondary
Teachers &
-Pr-inc iip-a-l-s
89
9 5
95
100
100
100
100
9 5
100
9 5
100
100
89
9 5
100
100
95
95
100
89
100
9 5
89
7 4
84
100
9 5
100
95
95
100
100
100
100
95
95
100
89
9 5
9 5
95
9 5
9 5
9 5
9 5
95.6
A1 1
Teachers &
-P r i ~ i cpi-a -l s
93
97
97
9 7
100
100
100
95
100
97
100
100
9 5
97
100
100
97
9 7
LOO
95
100
9 5
95
8 5
8 7
100
97
100
9 7
97
100
100
100
100
9 7
98
100
9 5
97
97
98
9 7
97
97
95
97.3
Teacher
Educato-rs-
100
100
100
9 4
94
100
100
88
9 4
8 7
94
94
9 4
94
8 7
94
9 4
100
9 4
87
7 5
100
8 1
7 5
81
100
81
100
100
9 4
100
100
100
8 7
8 7
8 7
6 9
6 9
81
9 4
9 4
100
8 7
94
3 7
91.4
TABLE 15. Percent of Different Ethnic Groups Who Felt That
Tqere Had Reen an Opportunity to Learn the S t i l l
Req~a ired by the Writing Flu1 t i p l e Choice Test Itein
I tern
-N-um-b er
4 5
Average
h e r i can
---I-n- dian
100
100
80
60
80
100
100
100
100
80
100
90
80
40
80
80
100
80
100
80
60
40
20
60
60
80
60
80
60
4 0
80
80
80
80
60
60
40
40
80
80
80
50
60
GO
60
Group
Black Caucasian Hisganic 7
100
100
100
9 1
9 1
100
100
9 1
100
9 1
100
100
52
73
180
100
100
100
9 1
9 1
64
82
73
73
64
100
91
9 1
9 1
91
100
100
100
82
100
91
73
64
64
9 1
9 9
100
82
82
82
Other
-E- thnic
100
100
100
50
100
100
100
100
5 0
0
100
50
108
100
50
100
50
100
5 8
100
100
5 0
100
100
100
100
100
100
100
100
100
100
100
50
100
5 0
53
0
0
5 0
5 0
50
0
0
0
TABLE 16. Percent o f Members w i t h D i f f e r i n g Involvement
i n Education Who F e l t That There Had Been an
Opportunity t o Learn the S k i l l Requit-?d by the
W r i t i n g M u l t i p l e Choice Test Item
I tem
Number p-
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2 0
2 1
22
23
24
2 5
2 6
2 7
28
2 9
30
3 1
32
3 3
3 4
35
36
37
3 8
39
4 0
4 1
42
4 3
44
45
Average
Elementary
Teachers &
-P-r i n c i p a l s
9 5
100
95
90
9 5
100
100
100
100
9 5
100
100
100
95
9 5
100
9 5
90
100
100
90
90
86
100
86
100
100
100
100
100
100
100
100
90
95
8 6
90
81
81
90
95
9 5
8 1
81
90
94.5
Group
Secondary
Teachers &
P r i n c i p a l s
89
89
84
84
100
100
9 5
8 9
95
84
100
8 9
84
79
7 4
89
89
79
100
7 9
7 9
79
58
6 3
74
89
68
89
7 9
84
89
89
89
68
84
84
7 4
58
7 4
79
7 4
84
5 8
74
74
81.9
Teacher
Educato--r s
94
88
9 4
81
9 4
9 4
100
81
94
6 9
94
88
6 9
88
9 4
100
88
9 4
88
6 9
5 9
81
63
5 0
6 9
100
75
100
100
88
88
88
9 4
81
88
88
56
44
5 0
88
9 4
94
7 5
6 9
69
82.7
I tern
-N-um ber
Average
TABLE 17.
% Very
-R--e--1 evant
Judgments of Panel Members wit+ Respect to
Relevancy and Opportunity to Learn for Each
Item on the Writing Multiple Choice Test
-- Re 1 e v an c y
% Very or
Moderately %Very, Moderately,
-Re 1 evant - or Sl igh--t ly R-e-le vant
Opport~nity
to Learn
% Yes -
95
9 3
93
8 2
96
98
9 6
9 1
94
82
100
93
88
82
9 1
98
93
8 6
96
8 2
7 7
84
7 1
70
7 5
9 6
84
9 6
9 3
9 1
9 3
93
93
80
88
7 9
73
61
7 0
7 7
88
9 1
7 1
7 5
7 7
Test
TABLE 18. Percent of Items Viewed by Specific Percentages
of Panel Members in Different Categories with
Respect to Relevancy and Opportunity to Learn
- - Re 1 e v an c y
Very or Very, Moderately
Percent Very Moderately or Slightly Opportuni ty
-of- J u a e-s Relevant Relevant Re1 evant --- - -- to Learn- -
Mathematics 50% or more
67% or more
75% or rnore
Read -i ng
@ Writing
Multinle
Choice
50% or nore
67% or more
75% or rnore
50% or more
67% or more
75% or more
TABLE 19. Percent 0.f Panelists in Various Ethnic rJroups
Who Feel Teachers or Students in a Teacher
Education Program Need to Be Able to Write an
Acceptable Essay and Who Feel They Have Had
an Opportunity to Learn to Write
Need to Be Able Had an Opportunity
-Gro up t o Write to Lear-n to- Writ e
American I nd i an 80 10D
Black 86 8 6
Caucasi an 87 80
Hispanic 82 82
Other
A1 1 Panel i s t s
TABLE 20. Percent of Panelists with Different Involvement
in Education Who Feel Teachers or Students in a
Teacher Education Program Need to Be Able to Write
an Acceptable Essay and Who Feel They Have Had an
Opportunity to Learn to Write
Group
Need to Be Able Had an Opportunity
t o Write t o Learn to Write
Elementary Teachers and Principals 86 7 6
Secondary Teachers and Principal s 7 4 79
All Teachers and Principals 80 7 8
Teacher Educators 100 9 4
A1 1 Panel i s t s
TABLE 21. Frequency Distribution of the Passing Scores
for All Three Multiple Choice Tests
Score
40
3 9
3 8
3 7
36
3 5
3 4
33
32
3 1
3 0
2 9
2 8
2 7
26
2 5
24
2 3
2 2
2 1
2 0
19
18
17
16
15
14
13
12
11
10
Number o f Panelists
-Mat hemat i cs Readillg -W- riting
1
0
0
1
0
6
2
2
4
2
1
5
4
2
4
4
1
2
1
4
2
2
0
1
1
1
0
2
0
0
1
TABLE 22. Average Passing Scores on the Multiple
Choice Tests for Various Groups
-G- roup Mathematics
American Indian
Black
Caucas i an
Hispanic
Other Ethnic
Elementary Teachers
and Principals 26
Secondary Teachers
and principals 23
A1 1 Teachers and Principals 25
Teacher Educators 2 6
A1 1 Panelists 2 5
Reading
TABLE 23. Average Passing Score on the Writing
Essay Question for Various Groups
Group
Amer i can I nd i an
51 ack
Caucasi an
Hispanic
Other Ethnic
Elementary Teachers and Principals
Secondary Teachers and Principal s
A1 1 Teachers and Principal s
Passing Score
Teacher Educators 9
A1 1 Panel ists 9
TABLE 24. Estimated Failure Rates for Three
Sets of Possible Passing Scores
TABLE 25. Possible Arizona Passing Scores and
Passing Scores Used in Other Places
APPENDIX A - - -- .- -
Test Specifications
R E Pre-Professional Skills Tests
Copyright @ 1982 by Educational Testing Service. A31 rights reserved.
Princeton, N.J. 08541
This t e s t merrurer a vide varlety of s k i l l s and ranges over a vide variety
of u t t r i a l s . h e materials do not c a l l on outside infomation but r e f l e c t mny
sscr:cr, issues, and l e v e l s of r t r i o u m e s s . In rirmplry, the rourcer for reading
passages and questions are =hat teacherr rarQ mil are axposed to.
The three mafor s k i l l areas, together with F a t t r p r e t i v r rubr"nills*, are as
fo?lovs:
The r b i l i t y to mderstand accurately and complcctrlly the e x p l i c i t
content of r v r i t t e n message.
1. #th I d a
2. Detail
e.g., Definition - word, phrase, s t c .
Supporting I d a s
3. Relatfocohipr
e. g., Sequence
Caure and Effect
I . Paraphrrse/S-rg of alementi in the merrrge
The a b i l i t y t o c l a r i f y a written message and underitand hov it
I s orianized and conveys its message.
The writer'@ purpose
The w r i t e r ' s assusptions
The b-ritar's a t t i t u d e o r tone
I n p l i c r t i o n s of the message
Inferences fron the message
t a c t vs. opinion in the massage
Orgaairaeiw of the message
Use of langaugc i n the message
Application of elcslents i n the message
C. Evaluation approx, 152 (7 t o 8 it-)
Phe ab?.lity t o make reasoned g u a l l t r t i v e j u d p e a t s aSout the nature
and mirits of a rritteri message.
P. b o t i o n o l or manipulative r r p e c t s of the message
2. Strengths and/or weaknesses of the argwaent
3. Relevance and/or appropriateness cf supporting
evidence, arguments
h. Relation of the message to the a u d i a c e and/or to
the general m i v e r s e of the topic
+here are three types of items iaeluded:
I, r long passage (200 words) srlth a r a t s f 5-7 Atems
XI. a short passage (100 words) with a r e t of 2-3 items
111. d i s c r e t e items with r b r i e f o t t u l u r
e
These oubskillr are rmpled for pur?oces of t e s t eonstruetion.
CO?':'?.!r;HT, EDUCATIONAL TESTING SERVICE PRINCETON, HE;; JERSEY 085::
Those tygrer are dirtrfbuted %sr arch t e s t form as foZlossr:
Bun'Ser of parsa~es: 2-3 4-3 -
Tptr?l Imbrr of quastionr: a4-17 Il-U U-i3
Percentage of taot: ogprox, 40% approx. 30X approx. 3CZ
f t r n s %All explore understanding of 8 range of u d t s of discourre (wrds,
phrases, claurer, eeatenceo, idea clusters, paragraphs, etc.). The p a e ~ a g e fo=ts
obvioorPy pemit testing of a broader range of units of d i r c o u r ~ et han does the
d i r c r e t r itas fomat.
Itmr in u c h of the three fowurts my pose q u ~ s t i o n so f v a r y h g d l f f i c u l ~ y
and t e s t aay of the a U l a .
?he content of arch form of the rest is to Be a0 foUovo:
Subject Matter:
1. Teacher-related apprax. 68%
P r o f e s r i ~ ~Caoln duct
Poldey lrtues
Philorophderl concerns
2. General Xnterasr approx. 10%
1. Tuo passages Ln each t a r t should deal with special concerns:
one with ~ o r i t y - r e l a t e d issues and one with lsrues of eon:er=l
to wotgcn.
S. At least one long passage vfll present m t a r i a l that it cxplicft
or rroily understood; the other(o) vhll prrsrslt a more coaplu
ddreursion.
2. Passage6 rhsuld u e a ~ l i f ya variety of modes of prose vrftfng, e.g.,
dircursive, descriptive, narrative. At l e a s t one passage ahodd be
of 8 technical or scientPfic ooture.
Total n d a r of it-: 60.
WRITING TEST SPECIFXCATIONS
This t e s t assumes t h a t teachers a r e informed c i t i z e n s of the world who
have a variety of concerns and i n t e r e s t s . Therefore, t h i s t e s t is not limited
t o matters dealing with the routine a c t i v i t i e s of the school day, although
r u b s t a n t i a l and serious educational issues are included.
The t e s t assumes t h a t an e f f e c t i v e v r i t e r should be able t o do the fol-lowing.
1. Provide and sustain a focus or t h e s i s .
2. Attain in d i f f e r e n t papers the varied aims o r purposes (e.g., explanatory,
persuasive, expressive) of discourse.
3. Dccide which of these aims or purposes is appropriate i n a given writing
s i t u a t i o n .
h . Select and s u s t a i n an appropriate persona or voice.
5. Produce and develop adequate and appropriate material t o accomplish the
purpose for w r i t i n g , identifying and supporting, as appropriate, impor-tant
assuslp t ions.
6. Choose and use a mode of organization c o n s i s t e n t l y .
7. Preserve coherence i n an extended piece of writing.
8. Choose an a7propriate mode or organization (chronological, enumerative,
e t c . ) .
9. Construct sentences i n standard written English, adjusting c\olcc of
sentence s t r u c t u r e and word choice to s u i t purposes and aims.
10. Use sentences and vocabulary vhich are appropriate to the pur?ose of r5e
writing.
11. Use words and sentences which are appropriate for the intended readers.
. Construct sentences in standard written English and i d e n t i f y sentences
that do not observe the conventions of standard written English, scch
as graanar, usage, and punctuation.
The f i r s t part of the t e s t cancains 65 multiple-choice i t e m of NO
types :
25 usage items
20 rcntence correction items
The usage item type asks for the correction t h a t atst be mde to a ccnte3ce i f
it is to meet the demands of Standard/Edited American English. The seatencs
correction item type arks the inee t o choose the most e f f e c t i v e way to
r e s t a t e (I given phrase or sentence. Both item types include "no error"
options; t h e r e are 7-9 items keyed "no error" in each t e s t form.
COPYRIGHT, EDUCATION& TESTING SERVICE
The folloving points are measured with both deem types:
A. Structure (Gr r and b g i c a l Relationships)
1. Noun, Pronoun, Verb, Adjective and Adverb Problans - 8 to 11 ite=s
2. Co~rd'nation, Subordination, Correlation, Comparison, Parallelis-,
and Negation Structures - 15 t o 21 items
B. Diction, Idiom, and Hechanics (Redundancy, Word Choice, Punctuation,
and Capitalization) - 8 t o 14 items
The second part of the t e s t c o n s i s t s of one reporately timed twenty-minute
writing sample. The stimulus for t h i s is t o be both credible and s u b s t a c t i a l ,
but not necessarily limited t o the f i e l d of educatlan. The assignnent cou:d
be very structured or open.
Total number of i t e m : 46 (45 multiple-choice and 1 essay)
The eornpetencier measured aro believed t o be developmmtal in nature, accruing
not by an inCividual Raving taken one or more opecific courses, but rather
through the e m u l a t i v e e f f e c t of tho t o t a l mathematier eurrjculwa. There is,
of course, an arrunptioa bere t h a t c e r t a i n comnonalitieo e x i s t i n sdl
mthematlcs curricula, and that a l l candidates vill have had c e r t a i n "basics";
for exaaple, it is assumed t h a t a l l candidates d P 1 b v e rtudiad f r a c t i o n s ,
regardless s f the c i t y or r t a t e i n which they went to rchool, or the textbook
o r methodology used In the school(s). Therefore, these eompethncits ore considered
t o be neither content specffjte nor eontrrrt f r s r .
e
+he r f x eomgetencies, together wtth i n t e r p r e t i v e subrkil1s , are as
f oPlous :
A. Ras good nw5er sense, t h a t is, understands haw numbers behave. (8 to 10 i t a s )
1. Bas a lens. of order among n=bers--r.8.. bows t h a t is bemeen f ant -2I * 3
t h a t -1 < -2. that 1.9 is e201er t o 2 than 2 : $8, t h a t 751 i1 l e s s than 1,
and that 600: is more thaa 1.
2. Ras r meaningful understanding of the vay nucbtrr are named, ( i . e . , place
value); understand6 that r number has many n o e s , and has f a c i l i t y i n
. t r a n s l a t i n g from one to another as needed; e.8.. can use 502 or .5 or f -
whichever o i m p ~ i f i r r o mp put at ion or aid8 f l e x i b i l i t y of thinking.
3. Elm@ 8 rense of the order of magnitude of n u ~ b e r41 s it r e l a t e s to place value
or r e i c n t i f i c notation; s.g., recognires t h a t 100 is 1,000 tines 8s great as
3 1 5 0.1. o r t h a t 2.57 x 10 1s 8s great as 2.57 x 10 .
I . Estimrter, ox o t h e ~ d i ~pree dicts, the outcome of computation.
1. Recognizes an appropriate match betvccn mathezatics and r e a l l i f e ;
r.gvr ean s s t n b l i t h r correct r a t i o or percent, o r s e l e c t an rpprepriace
operatSon for r r e a l l i f e problm.
2. Recognizes necessary and t u f f f c i e n t eondlt4onr f o r the r o l u t i o n of real l i f e
probleas; o.g., f o r r rema l i f e problrc;, knows vBat numbers . x i needed -and hoe.
to obtain fh'cm (what mcmsurlmcnts are needed, 8 . 8 . )
3. Selves r e a l l i f e problems by e r t i a c t i n g antiters and doing the necessary
computation..
4. RecognJtes and ehoosez multiple ways to find answers and equivalent
computationa4 procedures.
*
There r u b r k f l l r Bra r a ~ p l e d for purposes of t e s t construetion.
COPYRIGHT, EDUCATIONAL TESTING SERVICE PRINCFTOK, NEt: JERSEY 085L1
5. Recognizes an appropriate nwnber that can be used as an answer to a problem
and adjusts and interprets anovers to fit the context of tho problem;
e.6.. the answer, 3 -31 vould br recorded as 6 if it represent8 the number of
needed to transport people, but recorded 8s 5 if dt represent8 the
aumber sf pissennerp par car.
6. Correctly prodiets the outesme of changing 6aoe number or condition in a
problem in an "if-then" rense-r.g,, If N s 5 m Q, what $8 the value ef
1 4 101
7. Interprets numbers when used to express probability.
C. Recon~izes a>d uses usthezstical ~elstionshipr. (8 to 10 ftes)
1. Distizlgulshes mong direct, inverse and other kinds of variation without
arcessarily knowing the eorrret t e w for the relrtioolhip.
2. Recognizes spatial relationships in everyday life--@.go, identifies a d
predicts possible re1a:ionrhlps among l i ~ r rdn rpaea.
3. States and tqres relationships for ehe measurer eP eomzon two- md-three
dimensional geonetrie figures.
4. Syr.bolizes r relationship rpproprirtely; tonverrePy, interprets a ~elation~tip
expressed in syn5ols; understands the use of a Pornlo as a way to solve a
class of siEilar problems.
5. Recognizes equivalent relrtionshipr having a different foPm-t*t*r
d E r t - t - d t r .
6. Solves problers involving ratio and proportion and pereent.
7, Recognizes relationships evident in data and mikes appropriate predictions
and/or sxtr~polations Prom that data.
b. UnCcrstrnds the mathematical basis of measurement. (4 to 6 ig-)
1. Understands that numbers are used to quantify rttributcf (a.g., length,
temperature, nr~a) of ~bjeets, xaot the objects the~selve~.
2. lecognirta and uses appropriate unitr Isr =king everyday neasurements.
3. RecognSrea and uses geometric concepts in making linear, area, (PGB vofme
'D@BCUTG%~O~~*
I . Underrtands the relationship between the rite of the unit and the number of
units--e.g.. the rhorter the unit used ta measure length, the larger the
number of-thore units in a specific ocrrurmenc.
0 5. bovs in m general way hou.to convert from one unit to another in the
6oae system--i.e., whether to muftiply or divide.
6. Dettminer the otasurements needed in order to eolve m prob1un;'can rolve
measurement preblems.
7. f s literate about the metric (Sf) sj$tem.
8. Reres a calibrated rcale correctly, vhether the calibration it in multiples
of whole numbers er fractional division; estimates readings bervcen tic marks.
L. Understands deduetivc rersonfng. (4 to 6 it-)
1. Gerrectly interprets rentenceo which incorporate the logical eonneetiver,
"and", "or", and "if-then" ma well as the quantifierr, "some", "all", and "none".
2. Uses dedoc:lve reasoning to determine whether #J conclusion based on a
series of rtatements about everyday situatiom is valid or invalid.
3. Sees the need for barie definitlonr and aroumptions and recognizes hidden
ast+&ptioar--e.g., in advertiremento or pofiticml rfogans.
I. Fakes appropriate generalizations; identifier eounterexaqles to inappropriate
e F. gGeange rian%eiezrastrieotn sg.r aphic. s . r ~ , b o l i ca ac! verbal material. (4 to 6 itant)
1. bkcs reasonable visuah eomparisont ef the rite of 'two oe more objects.
2. Reads and intcrpretr bar, line and circle graphs and pictographs.
3. Chooses r ath he me tie ally appropriate gra2h to represent a given tet of data.
4. Interprets a schematic diagrm-e.g., m flw-chart, rlectrica2 wiring
diagram or diagrm of the circulatory ryrtcm of r frog.
Total number of item: 40.
APPENDIX B
ARIZONA BOARD OF REGENTS
EDUCATION BUILDING (602) 255-4082
1535 WEST JEFFERSON, PHOENIX, ARIZONA 85007
March 26, 1985
LETTER MAILED TO EACH DISTRICT SUPERINTENDENT
Dear
I n 1934 the Arizona L e g i s l a t u r e enacted a b i l l r e q u i r i n g applicants f o r
admission t o teacher education degree programs at the u n i v e r s i t i e s t o pass a
basic s k i l l s t e s t i n mathematics, reading, and grammar. The Board o f Regents
has decided t o use the Pre-Professional S k i l l s Tests (PPST) developed by the
Educational Testing Service t o s a t i s f y the s t a t u t o r y requirement. However,
before the t e s t s can be used, they must be p r o p e r l y validated f o r use i n
tlri zma.
The Board i s forming an Arizona V a l i d a t i o n Panel t h a t w i l l be comprised o f
3pproximately 40-50 members, i n c l u d i n g elementary and secondary teachers and
p r i n c i p a l s from Arizona schools and teacher educators from the u n i v e r s i t i e s .
Superintendents o f Arizona school d i s t r i c t s are being requested t o nominate
:naster teachers and p r i n c i p a l s t o serve on the Panel. From the l i s t of nomi-qees,
a group of teachers and p r i n c i p a l s r e p r e s e n t a t i v e of the Arizona popula-t
i o n w i l l be i n v i t e d t~ serve on the Panel.
The Panel will meet i n Phoenix on Friday, A p r i l 26, f o r a f u l l day. The
yl~rpose of the meeting w i l l be t o review the PPST t o determine whether it
neasures basic s k i l l s needed by teachers and t o a s s i s t i n s e t t i n g the minimum
passing score. The judgments t h a t w i l l be made by the Panel w i l l be c r u c i a l t o
the accomplishment of the o b j e c t i v e o f ensuring t h a t a l l students preparing t o
enter t h e teaching profession have mastered the relevant basic s k i l l s .
Mould you please a s s i s t us i n t h i s p r o j e c t by nominating master teachers
and p r i n c i p a l s from your d i s t r i c t who have demonstrated a high l e v e l of a b i l i t y .
It i s our hope t h a t the nominee pool w i l l i n c l u d e b o t h men and women and i n d i -
v i d u a l s from various e t h n i c groups. They should be s e n s i t i v e t o the needs o f
students from a l l c u l t u r a l backgrounds. We would also l i k e t o have representa-t
i v e s from various grade l e v e l s and from various teaching f i e l d s .
THE I!h:IVE9SITY OF ARIZONA
TUCSON, A91ZONA 85721
ARIZONA STATE UNIVERSITY
TEMPE, ARIZONA 85287
NORTHERN ARIZONA ONlVEHSlTY
FLAGSTAFF, ARIZONA 8601 1
March 26, 1985
Page Two
The Board of Regents will provide funds t o cover the travel expenses of the
participants from outside Maricopa County and the meal expenses for all partici-pants.
We hope the school districts will be willing to provide release time for
any of their teachers selected t o serve on the Panel.
Enclosed is a form on which the nominees from your district should be iden-tified.
Each elementary and each secondary district is invited to submit the
names of up to 10 teachers and 3 principals. Unified districts are invited to
submit the names of up to 10 elementary teachers, 10 secondary teachers, and 6
principals. The number of names submitted will vary according t o the size of
the district. In order to ensure that the Panel is representative of the
Arizona population, i t is important that we have all of the information
requested for each nominee. Please return the completed forms no later than
April 8 t o the following address:
Arizona Board of Regents
1535 West Jefferson, Suite 121
Phoenix, AZ 85007
Your assistance in identifying outstanding teachers and principal s t o
help set the standards for future teachers wi 11 be appreciated.
Sincerely,
Robert A. Huff
Executive Director
Esther N. Gapin
Regent
mbb
Encl osure
ARIZONA BOARD OF REGENTS
EDUCATION BUILDING (602) 255-4082
1535 WEST JEFFERSON, PHOENIX, ARIZONA 85007
March 28, 1985
LETTER MAILED TO EACH UNIVERSITY PRESIDENT
Dear
As we prepare for the implementation of the Pre-Professional Skills Test
(PPST) developed by ETS for the assessment of basic skills of candidates for
teacher education programs on the three campuses, we need to form a panel
i ncl udi ng teacher educators from our three faculties. The guide1 i nes from ETS
suggest that our panel should represent the demographics of the State of
Arizona. Thus, we will need to include both men and women as well as represen-tatives
from each of the ethnic minority groups in the state. I am asking you
t o nominate ten individuals from your teacher education faculty and include with
their names the following information: sex, ethnic origin, subjects taught, and
years of service at the university. From the list of nominees, we expect to
select approximately five faculty members from each university.
We will need your l i s t of nominees in the central office not later than
April 5, 1985. Should you have any questions about the validation process or
the use of the PPST, please feel free to call me or Dr. Elliott.
Thank you for your assistance with this important matter. We will keep you
i nforrned as the content val idation process proceeds.
Sincerely,
Robert A. Huff
Executive Director
THE UNIVERSITY OF ARIZONA
TUCSON, ARIZONA 85721
ARIZONA STATE UNIVERSITY
TEMPE, ARIZONA 85287
NORTHERN ARIZONA UNIVERSITY
FLAGSTAFF, ARIZONA 86011
-AP-P-EN-D IX C
Population of Arizona, By Racial/Ethnic Croup
Population of Percentage of
Racial/Ethnic Group Racial/Ethnic Group S t a t e Population
Black 74,977 2.8
American Indian,
Eskimo, and Aleut
Asian and P a c i f i c
I s l a n d e r
Spanish Origin 440,701 16.2
T o t a l
a Because i n d i v i d u a l s included i n the "Other" category of t h e 1980 Census
a r e not i d e n t i f i e d as m i n o r i t i e s , t h e above percentages underestimate the
minority population.
Source: U.S. Department of Commerce, Bureau of Census, 1980 Census of
Population, "General Population C h a r a c t e r i s t i c s , " pp. 4-122.
APPENDIX D
ARIZONA BOARD OF REGENTS
PRE-PROFESSIONAL SKILLS TESTS
VALIDATION STUDY
I d e n t i f i c a t i o n Information and General I n s t r u c t i o n s
f o r
Teachers and P r i n c i p a l s
-
( o f f i c e use only)
2. Gender (Check one) F M
3. E t h n i c i t y (Check one)
American I nd i an -9 Black -9 * Caucasian ; Hispanic y
Other (Specify)
4. Place of Employment
School Name ; D i s t r i c t Name --
5. Type of School Involvement (Check one)
Teacher -9 P r i n c i p a l ; Other (Specify)
6. Grade Level of Primary Involvement (Check one)
K-6 ; 7-12 ; Other (Specify)
7. Subject Matter Taught ( i f applicable)
ARIZONA BOARD OF REGENTS
PRE-PROFESSIONAL SKILLS TESTS
VAL I DATSXTUNDY
Genera1 Instructions for Multiple Choice Tests
for
Teachers and Pri nci pa1 s
For each item in this test you will need to make three (3) separate judgments:
1. The extent to which the knowledge or academic skill tested in the
question is relevant to competent performance as a teacher in Arizona;
2. Whether a typical applicant who graduated from an Arizona high school
and/or who has met the course prerequisites for admission t o a teacher
education program had an opportunity t o acquire the knowledge or skill
which is required to answer the test question;
3. The proportion of marginal11 qua1 ified applicants for a teacher educa-tion
program whom you would expect to answer the test questions
correctly.
In addition, each panel member should identify any questions that include
language that would result in a bias against a menher of an ethnic minority.
The question should be identified in the comments section and a detailed expla-nation
of the bias problem should be provided.
The specific scales and responses to be used are described on the next page.
ARIZONA BOARD OF REGENTS
-PR-E-P ROFESSIONAL- - SKILLS TESTS
-V-AL-ID-A-T ION STUDY
Description of Rating Scale & Responses
for
Teachers and Principals
1. In making the f i r s t judgment, rate -e-ac h -item o n the t e s t on the following
scale:
You are to judge the extent to which the knowledge or academic s k i l l tested
in the question i s relevant to competent performance as a teacher in the
schools of Arizona.
Mark your answer according to the following scale:
VERY Needed by 50% or more of the teachers or by some teachers
RELEVANT 50% or more of the tirne, and i s usually important to suc-cessf
ul performance.
MODERATELY Needed by 25 to 49% of the teachers or by some teachers
RELEVANT between 25 and 49% of the time, and often i s important to
successful performance.
SLIGHTLY Needed by 10 to 24% of the teachers or by some teachers
RELEVANT between 10 and 24% of the time; and when used i s , at best,
only occasionally important to successful performance.
NOT Needed for fewer than 10% of the teachers or by some
RELEVANT teachers less than 10% of the time; and when used, i s not
important even though sometimes helpful to successful
performance
2. In making the second judgment you will need to make one of the following
responses for -each -item:
Yes, the typical applicant has had an opportunity to learn the knowledge
or ski1 l required,
No, the typical applicant has NOT had an opportunity to learn the
knowledge or s k i l l required,
3. In making the third judgment, record how many, out of 100 marginally quali-fied
applicants for a teacher education program you think would be able to
answer the question correctly.
ARIZONA BOARD OF REGENTS
PRE- PROFESSIONAL SKILLS TESTS
VALIDATION STUDY
Identification Information and General Instructions
for
Teacher Educators
--
(office use only)
1. Name
2. Gender (Check one) F M
3. Ethnicity (Check one)
American I nd i an ; Black _____> Caucasian ; Hispanic -3
Other (Specify)
4. Institute of Employment
5. Title of Position (Check one)
Professor ; Associate Professor ; Assistant Professor 9
Other (Specify)
7. Courses Taught (if applicable)
ARIZONA BOARD OF REGENTS
PRE-PROFESSIONAL SKILLS TESTS
VAL1 DATION STUDY
General Instructions for Multiple Choice Tests
for
Teacher Educators
For each item in this test you will need t o make three (3) separate judgments:
1. The extent t o which the knowledge or academic skill tested i n the
question is relevant to competent performance as a student in a teacher
education program at your institution.
2. Whether a typical applicant who graduated from an Arizona high school
and/or who has met the course prerequisites for admission to a teacher
education program had an opportunity to acquire the knowledge or aca-demic
skill which is required to answer the test question;
3. The proportion of marginally qua1 ified applicants for a teacher educa-tion
program whom you would expect t o answer the test questions
correctly.
In addition, each panel member should identify any questions that may include
1 anguage that would result in a bias against a member of an ethnic minority.
The question should be identified in the comments section and a detailed expla-nation
of the bias problem should be provided.
The specific scales and responses to be used are described on the next page.
ARIZONA BOARD OF REGENTS
PRE-PROFESSIONAL SKILLS TES--T S
VAL1 DATION STUDY
Description o f Rating Scale & Responses
f o r
Teacher Educators
1. I n making the f i r s t judgment, r a t e -e-a ch -ite-m on the t e s t on the f o l l o w i n g
scale:
You are t o judge the extent t o which the knowledge or academic s k i l l tested
i n the question i s relevant t o competent performance as a student i n a
teacher education degree program offered by your U n i v e r s i t y .
Mark your answer according t o the f o l l o w i n g scale:
VERY Needed by 50% or more o f the students or by some students
RELEVANT 50% or more o f the time, and i s usually important t o suc-cessf
u l performance.
MODERATELY Needed by 25 t o 49% o f the students or by some students
RELEVANT between 25 and 49% o f the time, and often i s important t o
successful performance.
SLIGHTLY Needed by 10 t o 24% o f the students or by some students
RELEVANT between 10 and 24% o f the time; and when used i s , at best,
only occasionally important t o successful performance.
NOT Needed f o r fewer than 10% o f the students or by some
RELEVANT students less than 10% o f the time; and when used, i s not
important even though sometimes h e l p f u l t o successful
performance.
2. In making the second judgment you w i l l need t~ make one of the f o l l o w i n g
responses f o r -each -item:
Yes, the t y p i c a l applicant has had an opportunity t o learn the knowledge
or s k i l l required,
No, the t y p i c a l applicant has NOT had an opportunity t o l e a r n t h e
knowledge or s k i l l required,
3. In making the t h i r d judgment, record how many, out o f 100 marginal1.y q u a l i -
f ied app1 icants f o r a teacher education program you t h i n k would he able t o
answer the question c o r r e c t l y .
ARIZONA BOARD OF REGENTS
PRE-PROFESSIONAL S K I L L S TES-TS
VALIDATION STUDY
RESPONSE SHEET
FOR
MATHEMATICS TEST JUDGMENTS
I .D. Number
i( o f f i-c e use io nl y)
RESPONSE SHEET
FOR
MATHEMATICS TEST JUDGMENTS
(Continued)
I.D. Number
( o f f i c e use only)
~ L i
RESPONSE SHEET
FOR
MATHEMATICS TEST JUDGMENTS
(Continued)
I .D. Number
I( o f f i c e use only)
COMMENTS :
* D i f f i cul ty Level : Easy - P-Val ues range from 94.49 - 74.50
Average - P-Values range from 74.49 - 54.50
D i f f i c u l t - P-Values range from 54.49 - 34.50
Very D i f f i c u l t - P-Values range from 34.49 - 14.50
ARIZONA BOARD OF REGENTS
PEE-PROFESSIONAL S K I L L S TESTS
VALIDATION STUDY
RESPONSE SHEET
FOR
READING TEST JUDGMENTS
.- --
( o f f i c e use only)
RESPONSE SHEET
FOR ( o f f i c e use only)
READING TEST JUDGMENTS
(Continued)
RESPONSE SHEET
FOR
READING TEST JUDGMENTS
(Continued)
1.U. Number
( o f f i c e use o n l y )
Re1 evancy t o Competent Performance
* D i f t ~ c u l ~Lyev el: Easy - P-Values range from 94.49 - 74.50
Average - P-Val ues range from 74.49 - 54.50
D i f f i c u l t - P-Values range from 54.49 - 34.50
Very D i f f i c u l t - P-Values range from 34.49 - 14.50
ARIZONA BOARD OF REGENTS
-PR E-PROFESSIONAL S K I L L S TESTS
VALIDATION STUDY
RESPONSE SHEET
FOR
WRITING MULTIPLE CHOICE TEST JUDGMENTS
I . D . Number
I--(-o--f--f-iIc e u s e only)
RESPONSE SHEET
F OK ( o f f i c e use o n l y )
WRITING M U L T l P L E CHOICE TEST JUDGMENTS
(Continued)
RESPONSE SHEET
FOR
WRITING MULTIPLE CHOICE TEST JUDGMENTS
(Continued)
( o f f i c e use o n l y )
* D - i f f i c u l t y Level: Easy -- P-Values range from 94.49 - 74.50
Average - P-Values range from 74.49 - 54.50
D i f f i c u l t - P-Values range from 54.49 - 34.50
Very D i f f i c u l t - P-Values range from 34.49 - 14.50
ARIZONA BOARD OF REGENTS
PRE-PROFESSIONAL S K I L L S TE-S- TS
VAL I DATI ON STUDY
Response Sheet f o r Judgments about
W R I T I N G T E S T ESSAY QUESTIONS
f o r
Teachers and P r i n c i p a l s
----
I .D. Nurnber
( o f f i c e use o n l y )
1. Does a teacher i n Arizona need t o be able t o w r i t e an acceptable essay i n
order t o perform s u c c e s s f u l l y as a teacher? (Check one)
Yes I
2. Have the a p p l i c a n t s who graduated from an Arizona high school and/or who
have completed the course p r e r e q u i s i t e s f o r admission t o a teacher education
program had an o p p o r t u n i t y t o l e a r n t o w r i t e an acceptable essay? (Check
one).
Yes CII No
3. From among the twelve (12) essays you read s e l e c t two which you t h i n k repre-sent
the minimum l e v e l o f s k i l l t h a t a new teacher i n Arizona should
demonstrate. Check the numbers o f the two essays i n the box below.
Check two and o n l y two
ARIZONA BOARD OF REGENTS
PKE-PROFESSIONAL SKILLS TESTS
VALIDATION STUDY
Response Sheet f o r Judgements about
WRITlNG TEST ESSAY QUESTIONS
f o r
Teacher Educators
--- -
I.D. Number
r( o-f f i c e use oIn l y )
1. Does a student i n a teacher education program a t your u n i v e r s i t y need t o be
able t o w r i t e an acceptable essay i n order t o perform s u c c e s s f u l l y i n t h e
program? (Check one)
CJ Yes INo
0 2. have the appl i c a n t s who graduated from an Arizona high school and/or who
have completed the course p r e r e q u i s i t e s f o r admission t o a teacher education
degree program had an o p p o r t u n i t y Lo l e a r n t o w r i t e an acceptable essay?
(Check one)
(-1 Yes INO
3. From among the twelve (12) essays you read s e l e c t -two which you t h i n k repre-sent
the minimum l e v e l o f ski 11 t h a t an a p p l i c a n t f o r admission t o a teacher
education program should demonstrate. Check the numbers o f t h e two essays
i n the box below.
Check -tw-o and o n l y two
APPENDIX E
PPST VALIDATION PANEL, 25 APRIL 1985
A. I n t r o d u c t i-o n
1. Thank you f o r coming t o do a d i f f i c u l t but a b s o l u t e l y e s s e n t i a l task.
a. You may or may not be aware t h a t l a s t year the Arizona L e g i s l a t u r e
enacted a b i l l r e q u i r i n g applicants f o r admission t o teacher educa-t
i o n programs at t h e t h r e e s t a t e u n i v e r s i t i e s t o pass a -basic -s -k i l l s
t e s t i n mathernat i cs, reading, and grammar.
b. After considering many a l t e r n a t i v e s , the Board o f Regents decided t o
use the Pre-Professional S k i l l s Tests developed by the Educational
Testing Service t o s a t i s f y the s t a t u t o r y requirement.
c. Although t h i s t e s t has been validated in, and i s c u r r e n t l y used in,
a number o f other s t a t e s , we want t o be -su-re t h a t each of t h e t h r e e
t e s t s i s appropriate f o r use i n Arizona.
2. Consequently, you have been assembled t o help us accomplish t h i s by
making c e r t a i n judgments about -each - i-te m o f each t e s t .
a. I n i n v i t i n g you t o p a r t i c i p a t e , we have worked very hard t o be sure
t h a t a l l Arizona constituencies are a p p r o p r i a t e l y represented.
b. As a group, you represent:
- both genders;
- a1 1 major ethnic groups;
- r u r a l , suburban, and urban schools;
- elementary educators, secondary educators, and teacher
educators;
- and, a v a r i e t y o f subject matter f i e l d s .
c. Thus, we are counting on you, as members o f a respected panel, t o
make judgments (and they are NOT easy ones) i n a careful and pro-f
e s s i o n a l manner.
APPENDIX E PAGE 2
3. Plan for the Day
a. First, Dr. Richard Peterson, Senior Research Psychologist at ETS,
will give you some background on the tests and their development.
b. Next, I will go over the scales and rating forms to be used in
recording your judgments so that everyone understands clearly what
is to be done.
c. Following that, we will, as a group, practice the task on a sample
item and discuss the outcome to be sure that the basis for the
judgment is clear to everyone.
d. Once everyone knows what they are to do, you will be asked to make
and record your individual judgments about each multiple choice item
on every test.
e. A t about 12:15 p.m., we will adjourn for lunch in the Alumni Lounge
on the second floor of the Memorial Union Building.
f. We need to be reassembled here no later than 1: 15 p.m. to continue
with the next stage.
g. A t that tirne, we wi 11 discuss procedures for judging the -s.a m- of
responses t o the essay question on the writing test.
h. Finally, you will make and record your judgments about the essays.
i. Then, when all materials have been turned in and checked, you may
1 eave .
-T-r aining fo-- r Judgments on Multiple Choice Tests
1. First, we need t o make sure the packet you received when you came in is
complete. I t should contain:
a. Identification information and general instructions:
- for teachers and principals; or,
- for teacher educators.
APPENDIX E PAGE 3
b. This three-page document should contain:
(1) an information blank f o r you t o f i l l out;
( 2 ) a set o f genera-l i n s t r u c t i o n s ; and,
(3) a d e s c r i p t i o n o f the s p e c i f i c r a t i n g scale and responses.
c. If you are missing anything, r a i s e your hand and we w i l l be sure you
get what i s needed.
d. Next, the packet should contain response sheets ( c o l o r coded f o r
each m u l t i p l e choice t e s t ) on which you w i l l record your judgments
about each itern.
You should have one f o r :
- Reading - blue;
- W r i t i n g - green;
- Math - yellow.
e. F i n a l l y , there should be a response sheet f o r your judgments r e l a t e d
t o the essay w r i t i n g question which we w i l l work on a f t e r lunch, and
which, again, i s s l i g h t l y d i f f e r e n t f o r teachers and p r i n c i p a l s than
f o r teacher educators.
3. If everyone has everything, we are now ready t o f i l l out the I . D . i n f o r -
mat i on sheet.
a. Please p r i n t your name so t h a t it can be read e a s i l y .
(1) While your name i s NOT t o be w r i t t e n on the response sheets, we
have put i n a pre-assigned I.D. number.
( 2 ) This number i s on each response sheet so t h a t a11 response
sheets can l a t e r be matched.
b. F i l l out the r e s t of t h e i n f o r m a t i o n very c a r e f u l l y .
(1) We need t h i s information because it i s very importank t o the
process.
(2) It i s important because we need t o know whether an item i s seen
one way by p a n e l i s t s i n general but i n a d i f f e r e n t way by any
p a r t i c u l a r subgroup o f you.
APPENDIX E PAGE 4
PPST VALIDATION PANEL (CONTINUED)
(3) For example, we need to know whether elementary teachers view
an item differently from secondary teachers or whether panel-ists
from urban schools see an item in a different light than
those from rural schools.
4. Now, please read the general instruct-io-ns about the three judgments you
will make for each m ~ l t i p l ~ c h o i citee m, and then review the specific
rating scale and responses to be used.
a. Please look at me when you have finished because I would like to
make a few comments about the interpretation of the scales before we
work on the sample item.
b. [After everyone has a chance t o review the material .]
(1) Is there anyone who needs more time to complete their reading?
( 2 ) when all are done, go on.]
c. The f i r s t comment I want to make is that, in making your judgment,
you are to view the teaching task in i t s b-roa-dest sense.
(1) That is, a teacher is NOT just a person in a classroom in
charge of children and students,
( 2 ) B u t rather, a true professional who not only must guide
instruction according to the l atest research results,
(3) B u t who also must communicate with colleagues, administrators,
and parents,
(4) And who, additionally, must represent their profession to the
public at large.
d. The second comment refers t o the way relevancy is t o be viewed.
I t is seen as having three components:
(1) Wow many teachers might need the knowledge or s k i l l ;
( 2 ) How often they need the s k i l l , and
(3) How important the s k i l l i s when needed and used.
APPENDIX E PAGE 5
e. While you will want to keep a l l three things in mind,
(1) How -im. p ortant the skill is when needed is of primary concern.
(2) Of secondary concern is the number of teachers (or students in
teacher education) who will be required to use the s k i l l .
(3) And of least concern is the frequency with which the s k i l l i s
1 f kely to be used.
(4) Because, even though not used often, a s k i l l may be c r i t i c a l
when i t -is needed.
f . My third comment r e l a t e s t o the judgment you have to make and record
in the -l a s t c olumn of the multiple choice response sheet.
(1) Here, you will need to specify the number out of 100 marginally
qualified applicants you think will answer the item correctly.
(2) In making t h i s judgment, you are helping to set a minimum- stan-dard.
A score which a marginally qu-a-lif-ie d teacher, as a pro-f
essi onal , wi 11 meet.
(3) I t is important to note that we are NOT talking about marginal
applicants who may or may not make i t ,
(4) B u t about someone who will be in the profession as a teacher.
g. Finally, you will note that at the end of each multiple choice t e s t
response sheet form there is a place for comments.
(1) I t is here that you are to identify any t e s t item that may
include language that would result in a bias against a member
of an ethnic minority.
( 2 ) To be helpful, we must know both the item number and exactly
what wording is likely to produce the bias. I t would also be
helpful if you can suggest a change in wording which would
eliminate the problem.
h. So, the judgments are complex:
(1) View teachers as true professionals in the broadest sense, and
( 2 ) View relevancy as, primarily, how important the s k i l l is when
needed and, sicondari ly, on- how many' are 1 i kel y to use the
ski 11.
APPENDIX E PAGE 6 a PPST VALIDATION PANEL (CONTINUED)
(3) And, i n specifying the number out o f 100 who w i l l get the item
c o r r e c t , you are t o consider a hypothetical group o f marginal,
b u t q u a l i f i e d , professionals.
(4) Any questions now before we try the sample item?
5. Now, look at the response sheet f o r the Reading m u l t i p l e choice t e s t .
a. We w i l l work one sample item on t h i s t e s t and discuss it before
going on.
b. Now we w i l l pass out the sample item. Please read it, make your
judgments, and record them i n the row marked SAMPLE.
c. [Project sample on screen once everyone has a chance t o read, judge,
and record.]
d. Ask: How did it go?
(1) L e t ' s see how much agreement we have on t h i s sample.
( 2 ) How many saw it as:
- Very Relevant -- ?
- Moderately Re1 evant ?
- S l i g h t l y Relevant -- ?
- Not Relevant ?
(3) -If [there are1 widely discrepant r e s u l t s :
- Ask someone who said very relevant why they chose t h a t .
- Then ask someone who selected n o t r e l e v a n t why.
P
e. Ask about the opportunity t o learn.
(1) How many:
- Yes ?
- No ?
(2) -If discrepant, discuss reasons.
APPENDIX E TRANSPARENCY FOR OVERHEAD PROJECTOR
Directions: Each statement or passage i n t h i s test is followed by a
question or questions based on its content. After reading a statement or
passage, choose the best anewer to each question from among the five
choices given. Answer a l l questlona following a statement or passage on
the basis of what is stated or implied i n t h a t Btatement or passage.
Ths new hand-held "talking" spelling computer aska the user t o s p e l l
a word, which it c l e a r l y pronounces. To the user's typed response, it
then e i t h e r gives praise or suggests another t r y , For spelling practice
there may be n ~ t h i n gb etter. For teaching spelling there may be nothing
worse, since few, i f any, of the programs for t h i s device provide instruc-tion
i n an order that exposes the patterns i n English epellirq.
Left to learn spelling by rote, children w i l l be unipble LO detect
or predict the regularitieo that characterize t h e i r written language.
They w i l l be ill-prepared to reason about a& choose correctly mong the
many options available for translating e p ~ e c he ounde into written repre-sentation.
Admittedly, spelling i n s t r u c t i o n i s o l a t e d from the logic of the
language still takes place i n some classrooms. And something can be said
for electronic evaluations that urge a aecosrd t r i a l before they correct
and fade, unlike single, written t r i a l s graded i n the pemaaance of red a ink. But for a l l t h e i r mechanized patience, feedback, and reinforcement,
spelling computers cannot replace the teacher. For mstery i n spelling
comes not only from learning which spellings are right and which are
wrong; it also comes from learning why they are so.
1. The main idea of the passage is that
(A) a child can perfect spelling s k i l l s only through practice
(B) the new spelling computers are limited i n t h e i r a b i l i t y to teach
children to s p e l l correctly
(6) classrom methods for spelling instruction need to be Improved
(D) patience, feedback, and reinforcement are the key t o e f f e c t i v e
teaching
(E) the new spelling computers provide one of the best possible £oms
of spelling practice
Educational Tes tdeg Service is an Equal Opportuni ty/
Affirmative Action Employer.
Copyright 1981 by Educational Testing Service. A l l rights reserved.
APPENDIX E PAGE 7
-PPS T VAL1 DATION PANEL (CONTINU-E-D )
f . Ask about the number in 100 marginally gualified professionals who
would get correct:
(1) If discrepant, discuss.
(2) Remember, we are setting a minimum standard for marginally
qualified professionals.
g. Any further coniments or questions?
(1) Okay. We are ready t o make individual judgments for a l l
multiple choice items on all three multiple choice tests.
(2) We will NOT go through a sample for the other tests.
(3) However, we ask that you work through them in order: Reading,
Writing, and Mathernatics.
(a) You may find id easier to make a1 1 judgments on one item
as you did with the sample.
(b) I t requires less time because you don't have to reread
the item for every judgment.
(4) Then, after lunch, we will go through directions for the essay
questi ons .
h. We will pass out all multiple choice tests now. Be sure you get one
copy of each.
(1) Since these tests are those used with actual candidates, the
security of the items is very, very important.
( 2 ) Consequently, we must ask that:
- You take NO NOTES.
- You NOT remove the booklets from this room.
(3) We wi 11 need to collect and account for them before anyone goes
to lunch.
(4) And, similarly, we will need to have you check them in with us
before you leave for the day if you need one back after lunch.
APPENDIX E PAGE 8
C. Training f o r Judgm-e-n ts on the W r i t i n g Essay Question
1. Hello again. T hope everyone enjoyed t h e i r lunch and i s now ready t o
get r i g h t t o the next judgment task.
2. Please take out and review the response sheet f o r judgments about the
w r i t i n g t e s t essay question.
3. The f i r s t two questions:
a. About the relevancy o f w r i t i n g ,
b. And about the opportunity t o learn t o w r i t e ,
c. Are s t r a i g h t f o r w a r d and s i m i l a r t o what you d i d on the m u l t i p l e
choice t e s t .
4. So, i t ' s the t h i r d t h a t I need t o discuss a l i t t l e .
a. As you probably surmised, you are asked t o read twelve essays w r i t -
ten by r e a l examinees i n response t o the w r i t i n g essay question.
b. Note t h a t each essay has i t s own code number.
c. After you have read -a l l -twe-lv e, please s e l e c t the -two which you
t h i n k represent the minimal l e v e l of s k i l l you t h i n k appropriate f o r
a new teacher i n Arizona.
d. And put an "X" i n the two boxes labeled w i t h the code number of
those two essays you have selected.
e. Note the boxes are labeled i n the same order as the essays i n your
packet, which w i l l be passed out now.
f. Again, remember your judgments are t o be made i n viewing the
teacher's task i n the broadest sense - as a professional- who guides
students, who communicates w i t h col leagues and parents, and who
represents the profession t o the public.
5. Any questions about the judgments regarding the essays?
a. Okay. Those f i n i s h e d w i t h the three m u l t i p l e choice t e s t s go r i g h t
ahead and complete your work f o r today.
APPENDIX E PAGE 9
@ -I'P-ST- V-A L1 DATION PANEL (-CO NTINUED
b. You may leave when:
(1) A l l task have been completed.
( 2 ) All materi a1 s given t o you have been returned.
c. If you did not f i n i s h the m u l t i p l e choice t e s t s before lunch,
please
(1) Come forward t o pick up and sign out f o r the b o o k l e t ( s ) you
need.
(2) Remember, we want you t o complete t h a t task before you s t a r t
reading the essays.
6. To a l l of you:
Thanks once more for p a r t i c i p a t i n g i n t h i s very important v a l i d a t i o n
process.
APPENDIX F
Guidelines Used to Score PPST Essays
Trdnsd redem rasign scores ba.ed an the following eC0ring guide. The ratings of two reders ere auaased to
produce thc onmay more. If the two ratings diugree by .ore than om point, the emsay i a r e d by a third
mder. A l l dirrspanciea in rating easayr are rrsclvsd bsfore PPST r o r e repopto me n l e d . There ir no
rescore a r v i c e available ?or the esray rection of the PET Writing laat.
Rat inq
6 pA 6a eras ieny thdeisao cnrsttwrootreya: a high dbprw of conptence in writing, though it my have rinor errors. A
e is m l l organized md well developed;
e urns apprapriote details to eupport a thesis or illustrate ideas;
e h w s mity, coherence, and prqreesion;
e declo~trrteso y n t r t i c variety; and
0 displsys clear facility in the use of language.
5 tAh 5i se sclastaeyg ocrlye:a rly dsmonstrates canpetence in writinq, though it may have minor errors. A paper in
o is well organized and well developed, though it may have fever details than does a 6 paper;
e ahom mity, coherence, a d progression;
a demonstrates -e a y n t ~ t i cv ariety; and
l displaye facility in language, tbugh it ray not be 8s flusnt ma r 6 paper.
4 tAh 4i se cssaatsyp odrsysa: onstrcltee competence in *ritJng, though it may have occasional errors. A paper in
o is dequately organized and dsvelap$d;
o usas m e details to eupprt a thesis o r i l l u s t r a t e ideas;
e Qsmonetrntee dequate facility with language; and
o may contain occasionall writing errors, but they are neither serious nor frequent.
3 cA e3t qsoPlrsyay r emvaye adlsim oonnst orra tme omre eof c tmhpee tfeonllcoew iinng wnrimtinsga, rbsu:t it is clearly flmwed. A paper in thia
a failure to oupg~ort a thesis or i l l w t r a t e generolizatione with .pprqpriate &tail;
0 lmk or variety in mntenca structure;
e limited or i n v p r q r i a t e word choice; and/or
e a prttern on r c u u l r t i c n of errors in anice, usage, or enntance structure.
0 dfmrpsnirrtton or very l i t t l e d.volaprent8
o l i t t l e or ,no M a i l or Lrmlavmt cpscif'ica; and/or
a wriars e r r a n in
1 A 1 eoscly *ntratea incolpet.nce In nititq. A r in thin category:
0 EonCo* ~ariwtrv d p m i 8 t m t writinp srm~.;en d
e may aXeo be i l l o g b a l , ineohd-mt, or mvsra9y underdeveloped. .
0 A pyar In tDaio ertqory Is written on m topic other thcn t)r cns ypcifial.
CopytQht 0 1984 By fducationpl 'leating Service. All rights remervd. 2/22/86