Performance Audit, Arizona Family Literacy Program

              State of Arizona
Office
of the
Auditor General
PERFORMANCE AUDIT
Report to the Arizona Legislature
By Debra K. Davenport
Auditor General
December 2000
Report No. 00-22
ARIZONA
FAMILY LITERACY
PROGRAM
The Auditor General is appointed by the Joint Legislative Audit Committee, a bipartisan committee
composed of five senators and five representatives. Her mission is to provide independent and impar-tial
information and specific recommendations to improve the operations of state and local government
entities. To this end, she provides financial audits and accounting services to the state and political
subdivisions and performance audits of state agencies and the programs they administer.
The Joint Legislative Audit Committee
Representative Roberta L. Voss, Chairman
Senator Tom Smith, Vice-Chairman
Representative Robert Burns Senator Keith Bee
Representative Ken Cheuvront Senator Herb Guenther
Representative Andy Nichols Senator Darden Hamilton
Representative Barry Wong Senator Pete Rios
Representative Jeff Groscost Senator Brenda Burns
(ex-officio) (ex-officio)
Audit Staff
Carol Cullen—Manager
and Contact Person (602) 553-0333
Tara Lennon—Team Leader
JoAnne Dukeshire—Team Member
Copies of the Auditor General’s reports are free.
You may request them by contacting us at:
Office of the Auditor General
2910 N. 44th Street, Suite 410
Phoenix, AZ 85018
(602) 553-0333
Additionally, many of our reports can be found in electronic format at:
www.auditorgen.state.az.us
2910 NORTH 44th STREET • SUITE 410 • PHOENIX, ARIZONA 85018 • (602) 553-0333 • FAX (602) 553-0051
DEBRA K. DAVENPORT, CPA
AUDITOR GENERAL
STATE OF ARIZONA
OFFICE OF THE
AUDITOR GENERAL
December 29, 2000
Members of the Arizona Legislature
The Honorable Jane Dee Hull, Governor
Ms. Lisa Graham Keegan,
Superintendent of Public Instruction
Department of Education
Transmitted herewith is a report of the Auditor General, an evaluation of Arizona’s
Family Literacy Program. This is the fifth evaluation and was conducted pursuant to
the provisions of A.R.S. §41-1279.08. I am also transmitting with this report a copy of
the Report Highlights for this evaluation to provide a quick summary for your
convenience.
As outlined in its response, the Arizona Department of Education (ADE) agrees with all
of the findings. However, ADE indicates that implementing the recommendations to
continue monitoring program sites is contingent upon receiving additional state
funding. Although nothing precludes ADE from doing so, it has chosen not to use any
of the program’s current funding for monitoring.
My staff and I will be pleased to discuss or clarify items in the report.
This report will be released to the public on January 2, 2001.
Sincerely,
Debbie Davenport
Auditor General
Enclosure
i
OFFICE OF THE AUDITOR GENERAL
SUMMARY
The Office of the Auditor General has conducted its annual
evaluation of the Arizona Family Literacy Program. The Legisla-ture
has required evaluations since the program was established
in 1994. Pursuant to the provisions of Arizona Revised Statutes
(A.R.S.) §41-1279.08, this evaluation provides information about
the program, recommends ways to improve the program’s ad-ministration,
and examines various aspects of the program’s
effectiveness.
The Family Literacy Program is directed at economically and
educationally disadvantaged parents with preschool-age chil-dren.
Family Literacy’s approach integrates adult, child, and
parent education in an effort to be more effective than programs
that focus exclusively on adults or exclusively on children.
The State Board of Education and the Arizona Department of
Education (ADE) are responsible for administering the Family
Literacy Program. Currently, ADE contracts with 13 providers to
operate 23 sites in 5 counties—Maricopa, Pima, Cochise, Cocon-ino,
and Yuma. In order to serve populations “most in need,” the
ADE only awards contracts to providers operating sites that
serve economically at-risk populations. These providers include
school districts, an adult education program, a community col-lege,
and community-based organizations. The Legislature has
continued to appropriate $1 million for the Family Literacy Pro-gram
for each fiscal year between 1996 and 2001.
ADE Improves Program
Administration and Sites Follow
Statutory and Model Program Guidelines
(See pages 13 through 19)
ADE’s Adult Education Division has improved its administra-tion
of the Family Literacy Program but needs to keep building
on these improvements. Last year’s evaluation highlighted the
need for better administration when it reported that 7 of 23
Summary
ii
OFFICE OF THE AUDITOR GENERAL
sites were not complying with basic requirements for the num-ber
of families enrolled or the implementation of program com-ponents.
In response, ADE hired a coordinator to make struc-tured
site visits to all program sites, monitor sites that were pre-viously
out of compliance, and maintain contact with site direc-tors
and program staff. ADE also improved its coordination with
two model program sites that provide training and technical
assistance to all program sites.
Previous OAG evaluations have found that site compliance with
statutory and programmatic requirements have fluctuated with
ADE’s intensity of administrative oversight. In years when ADE
improved administrative oversight of the program, compliance
improved. During the 1999-2000 school year, all Family Literacy
program sites operated in compliance with statutory require-ments
and program goals.
While all sites were in compliance with statutory and program
guidelines, some sites did not follow contractual guidelines for
administering tests and reporting results. To improve the accu-racy
and timeliness of data reporting ADE should continue to
develop the reporting systems, change the reporting require-ments,
and enforce data submission deadlines.
Adult Participants Make Gains
in Language Proficiency and
Employment Outcomes
(See pages 21 through 27)
During the 1999-2000 school year, Family Literacy program adult
participants made gains in developing their education skills and
joining the workforce. When adult participants enter the Family
Literacy Program, they enroll in one or both of two educational
tracks: the English for Speakers of Other Languages (ESOL) track
for English language development and the Adult Basic Educa-tion
(ABE) track for instruction in basic education topics, includ-ing
General Education Development (GED) preparation. Family
Literacy program adult participants who took ESOL classes in
the 1999-2000 school year made significant gains in their reading,
writing, and listening comprehension skills. Their gains in writ-ing
and listening comprehension skills were comparable to and
Summary
iii
OFFICE OF THE AUDITOR GENERAL
their reading gains were greater than those of participants in the
federally funded Even Start family literacy program.
Family Literacy program adults who took the adult basic educa-tion
track of the Family Literacy Program also made progress in
educational attainment, such as passing all portions of the GED.
However, on average, participants did not improve in their
scores on adult basic education tests. These results were similar
to those of the participants in the Even Start program.
In addition, Family Literacy and Even Start program adult par-ticipants
reported similar gains in employment. During the 1999-
2000 school year, gains were made in participants’ entry into the
workforce, entry into job training programs, and attainment of
certification necessary for certain employment.
Adult Participants Show Improvements
in Parenting Attitudes and Behaviors
(See pages 29 through 32)
Adults enrolled in the parenting skills component of the Family
Literacy Program improved both their attitudes about parenting
and their parenting behaviors. Most Family Literacy program
adults entered the program with positive parenting attitudes and
did not make large improvements to their attitudes. However,
Family Literacy program adults reported significant increases in
the number of times they demonstrated positive parenting be-haviors
at home and in activities associated with their children’s
school. A comparison of gains made by participants in the Even
Start program showed that while both programs had essentially
the same impacts on parenting behavior, the Even Start program
had a greater impact on parenting attitudes. These results were
similar to those found in the evaluation for the 1998-1999 school
year (see Auditor General Report No. 00-03, issued in March
2000).
Summary
iv
OFFICE OF THE AUDITOR GENERAL
Children in Family Literacy Improve
Kindergarten Readiness Skills
(See pages 33 through 38)
During the 1999-2000 school year, children in the Family Literacy
Program improved their readiness to succeed in kindergarten.
Analyses of program children’s scores on kindergarten readiness
assessments showed, that on average, program children in-creased
their development by 7 to 10 months. These results were
comparable to those made by children in the Even Start program
and to those made by children in the state-funded Early Child-hood
Block Grant (ECBG) preschool program. The latter is a
stand-alone preschool program rather than a family literacy
program. While only a portion of the Family Literacy program’s
4-year-olds advanced to kindergarten readiness levels in all test
measures, these results were not only consistent with the Even
Start and ECBG groups, but also with other research on similar
at-risk populations. Several factors associated with the program,
such as additional parenting classes and home visits by program
staff, were related to children’s levels of improvement. ADE
should continue to monitor implementation of model program
recommendations including those factors associated with im-proved
kindergarten readiness.
v
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS
Page
Introduction and Background.......................... 1
Finding I: ADE Improves Program
Administration and Sites
Follow Statutory and
Model Program Guidelines.......................... 13
ADE Improved Program
Oversight and Administration....................................... 13
All Sites Operate According to
Statutory Requirements
and Program Guidelines ................................................ 15
Continued Administrative
Oversight Needed to
Ensure Compliance......................................................... 16
Further Improvements
Needed at Site Level....................................................... 16
Recommendations .......................................................... 19
Finding II: Adult Participants Make
Gains in Language Proficiency
and Employment Outcomes........................ 21
Program Provides Two
Educational Tracks: English
Proficiency and Adult Basic Education......................... 21
Participants Improve
English Proficiency ......................................................... 21
Participants Show Gains in
Educational Attainment, But Not
in Tests of Adult Basic Education................................... 24
Table of Contents
vi
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS (Cont’d)
Page
Finding II: (Concl’d)
Employment Outcomes
Similar for Both Programs.............................................. 25
Finding III: Adult Participants Show
Improvements in Parenting
Attitudes and Behaviors.............................. 29
Program Seeks to Improve
Both Attitudes and Behaviors ........................................ 29
Parenting Attitudes Improve,
But Not as Much as Those of
Even Start Participants.................................................... 30
Adults Increase Positive
Parenting Behaviors........................................................ 31
Finding IV: Children in Family Literacy
Improve Kindergarten
Readiness Skills........................................... 33
Program’s Preschool Education
Component Aims to Enhance
Kindergarten Readiness ................................................. 33
Family Literacy Improvements
Were Similar to Comparison
Groups’ Improvements .................................................. 34
Table of Contents
vii
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS (Cont’d)
Page
Finding IV: (Concl’d)
Not All Program 4-Year-Olds
Advance to Kindergarten
Readiness Levels............................................................. 36
Program Can Build on Factors
Related to Children’s
Developmental Gains ..................................................... 37
Recommendation............................................................ 38
Appendix ........................................................... a-i
Agency Response
Tables
Table 1 Family Literacy Program
County, Contractor, Program Site, and
Number of Families Served
1999-2000 School Year
(Unaudited)................................................... 7
Table 2 Family Literacy Program
Schedule of Contractors and Contract Awards
Years Ended or Ending June 30, 1999, 2000,
and 2001 (Unaudited)................................... 9
Table 3 Family Literacy Program
Compared to Even Start and Early Childhood
Block Grant Programs
Participant Outcomes
1996-97, 1998-99, and 1999-00
School Years.................................................. 12
Table of Contents
viii
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS (Cont’d)
Page
Tables (Cont’d)
Table 4 Family Literacy Program
ADE Oversight and Site Compliance
1996 through 2000 School Years................... 17
Table 5 Family Literacy Program
Compared to Even Start Program
Changes in Frequency of Parenting
Behaviors
1999-2000 School Year................................... 32
Table 6 Family Literacy Program
Number and Percentage of Families
Completing Program by Site
1999-2000 School Year
(Unaudited)................................................... a-ii
Table 7 Family Literacy Program
Site Data Reporting Results and
Rankings by Improvements on Test Scores
1999-2000 School Year................................... a-v
Table 8 Family Literacy Program
Compared to Even Start Program
Average Improvements in English
Language Assessments
1999-2000 School Year................................... a-vi
Table 9 Family Literacy Program
Compared to Even Start Program
Average Scores on Tests of Adult
Basic Education
1999-2000 School Year................................... a-vii
Table of Contents
ix
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS (Cont’d)
Page
Tables (Concl’d)
Table 10 Family Literacy Program
Compared to Even Start Program
Average Improvements in Parenting
Attitudes
1999-2000 School Year................................... a-ix
Table 11 Family Literacy Program
Pre-K Success
Compared to Even Start and
Early Childhood Block Grant Programs
Average Improvements in Pre-K Success
Test Scores Converted to Months
1999-2000 School Year................................... a-x
Table 12 Family Literacy Program
Compared to Even Start and Early
Childhood Block Grant Programs
Number and Percentages of 4-Year-Olds
Who Met Kindergarten Readiness Levels
1999-2000 School Year................................... a-xi
Figures
Figure 1 Family Literacy Program
Annual Family Income Level
1999-2000 School Year................................... 3
Figure 2 Family Literacy Program
Adult Participants’ Family Status
1999-2000 School Year................................... 3
Figure 3 Family Literacy Program
Children’s Ages at Start of Program
1999-2000 School Year................................... 4
Table of Contents
x
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS (Concl’d)
Page
Figures (Concl’d)
Figure 4 Family Literacy Program
Participant Enrollment in Adult
Education Classes
1999-2000 School Year................................... 22
Figure 5 Family Literacy Program
Improvement in Language Assessment
System Test Scores
1999-2000 School Year................................... 23
Figure 6 Family Literacy Program
Compared to Even Start Program
Participants’ Employment Status at Exit
1999-2000 and 1998 and 1999
School Years.................................................. 27
Figure 7 Family Literacy Program
Average Improvement in Developmental
Skills Test Scores
Converted to Months
1999-2000 School Year................................... 35
1
OFFICE OF THE AUDITOR GENERAL
INTRODUCTION AND BACKGROUND
The Office of the Auditor General has conducted its annual
evaluation of the Arizona Family Literacy Program. The Legisla-ture
has required evaluations since the program was established
in 1994. This year’s evaluation provides information about the
program, examines various aspects of the program’s effective-ness,
and recommends ways to improve the program’s admini-stration.
Family Literacy Program Aims to
Improve Educational Skills of
Parents and Children
The Family Literacy Program aims to break the intergenerational
cycle of illiteracy by increasing the educational skills of children
and their parents. Children’s levels of literacy has been linked to
those of their parents. Similarly, parents’ involvement with their
children’s education has been linked to their education level. In
1999, the National Center for Education Statistics reported that
only 61 percent of the children whose mothers had less than a
high school education were read to three or more times a week.
For children whose mother’s highest level of education was a
college undergraduate, graduate, or professional degree, the
figure was more than 90 percent.
The greatest predictor of a child’s future academic suc-cess
is the literacy of the child’s mother.1
By breaking this cycle of illiteracy, the program also aims to re-duce
poverty in the next generation of families. Poverty has been
linked to both the parent’s level of education and the educational
opportunities for the children. Children whose parents lack a
1 Kerka, Sandra. Women, Work, and Literacy. Educational Resources Informa-tion
Center, Digest No. 92, 1989.
Introduction and Background
2
OFFICE OF THE AUDITOR GENERAL
By statute, to be eligible for the
Family Literacy Program, par-ents
must:
Ø Have a 3- or 4-year-old
child;
Ø Lack sufficient mastery of
basic educational or basic
English language skills to
function effectively in soci-ety,
or lack a high school
diploma or its equivalent;
and
Ø Be U.S. or legal residents, or
otherwise lawfully present
in this country.
high school diploma are more than twice as likely to live in pov-erty
than children whose parents are high school graduates.1
Program Serves Disadvantaged
Parents with Preschool Children
The Family Literacy Program is directed at economically and
educationally disadvantaged parents with preschool-age chil-dren.
The numbers, characteristics, and goals of the enrolled
participants are as follows:
n Families—During the
1999-2000 school year, the
program served a total of
391 families during all or
part of the year. A typical
family served had a family
income of less than
$15,000, spoke Spanish as
their primary language,
and consisted of an
unemployed adult who
participated in the pro-gram
with his/her
preschooler. Eighty-three
percent of the families
reported Spanish as the
primary language spoken at home. Eighty-three percent of
the families had three children or less and 56 percent had two
or less. As seen in Figures 1 and 2 (see page 3), 59 percent of
the families had incomes of $15,000 or less and 73 percent of
the families consisted of a couple with children.
1 National Center for Children in Poverty. Young Children in Poverty: A
Statistical Update, June 1999 Edition, New York: National Center For Chil-dren
in Poverty, 1999.
Introduction and Background
3
OFFICE OF THE AUDITOR GENERAL
Single parent
13%
Member of extended
family (not parent)
15%
Parent, part of a
couple with children
72%
Figure 2
Family Literacy Program
Adult Participants’ Family Status
1999-2000 School Year
Note: Of the 391 families served by the Family Literacy Program, 389 provided information for this question.
Source: Auditor General staff analysis of data provided by Arizona Department of Education and Family Literacy
program staff.
Non-parent family
members (14%)
Single parent (13%)
Figure 1
Family Literacy Program
Annual Family Income Level
1999-2000 School Year
Note: Of the 391 families served by the Family Literacy Program, 343 provided information for this question.
Source: Auditor General staff analysis of data provided by Arizona Department of Education and Family Literacy
program staff.
Over $20,001 (16%)
$15,001 to $20,000
(25%)
$5,001 to $10,000
(15%)
$10,001 to $15,000
(26%)
Under $5,000
(18%)
Parent, part of a
couple (73%)
Introduction and Background
4
OFFICE OF THE AUDITOR GENERAL
n Adult participants—Ninety-six percent of the adult partici-pants
were women. When the adult participants were asked
what goals they wanted to pursue, 101 indicated that they
were pursuing their General Educational Development
(GED) certificate and 280 said they wanted to improve their
English language skills. Sixty-two (17 percent) were em-ployed
at the start of the program: 27 working full-time and
32 working part-time. The median age was 28, with ages
ranging from 14 to 57.
n Children—Fifty-one percent of the children in the program
were boys and 49 percent were girls. As seen in Figure 3, the
majority of the children were 4 years old at the start of the
program.
Program Model Integrates the
Education of Adults and Children
Family Literacy’s approach integrates adult, child, and parent
education in an effort to be more effective than programs that
focus exclusively on adults or exclusively on children. Arizona’s
Family Literacy Program uses a model of family literacy advo-cated
by the National Center for Family Literacy (NCFL). Fami-lies
receive information and services in a classroom setting, gen-
Figure 3
Family Literacy Program
Children’s Ages at Start of Program
1999-2000 School Year
Note: All of the 391 families served by the Family Literacy Program provided information for this question.
Source: Auditor General staff analysis of data provided by Arizona Department of Education and Family Literacy Pro-gram
staff.
Over 4 years
old (6%)
Under 3 years
old (1%)
4 years old (61%) 3 years old (32%)
Introduction and Background
5
OFFICE OF THE AUDITOR GENERAL
erally on a school-year basis.1 The model integrates the curricula
and training found in the following four components:
n Adult literacy instruction—based on their individual needs,
adults receive instruction in English for Speakers of Other
Languages (ESOL) and/or Adult Basic Education (ABE)
which can include GED preparation.
n Early childhood instruction—preschool children receive
instruction from a developmentally appropriate curriculum
designed to improve their kindergarten readiness, specifi-cally
in the areas of cognitive, physical, and social skills.
n Parent and Child Together (PACT)—a parenting education
component in which parents and children learn to play to-gether.
Program staff help parents to understand the early
childhood learning processes at work in their children’s play
activities and to communicate with their children using posi-tive
parenting skills.
n Parent Time—provides opportunities for parents to learn
from the experiences of their peers, receive encouragement
from the group, and practice collective problem solving.
Program sites implementing this family literacy model can re-ceive
technical assistance from the NCFL through its Web site,
newsletter, training materials, and training sessions run by
NCFL national trainers. Additionally, Arizona’s Family Literacy
Program funds two model program sites, the Family Tree Project
of Mesa Unified School District and the Family Literacy Project
of Pima County Adult Education, to train Family Literacy staff
and instructors in the NCFL model. While state-funded family
literacy programs in other states pay for single NCFL training
sessions, Arizona provides ongoing technical assistance and
NCFL-certified training in program implementation, staff devel-opment,
and test administration throughout the year.
1 Sites provided families with informational materials such as instructional
videos on parenting, brochures on community services, children’s books,
and books on children’s development. Auditor General staff have no rec-ommendations
for additional informational material at this time.
Throughout the year, model
program staff provide tech-nical
assistance to Family
Literacy sites.
Introduction and Background
6
OFFICE OF THE AUDITOR GENERAL
Legislature Appropriates
$1 Million for Program Contractors
to Operate 23 Sites
The State Board of Education and the Arizona Department of
Education (ADE) are responsible for administering the Family
Literacy Program. In order to serve populations “most in need,”
the ADE only awards contracts to providers operating sites that
serve economically at-risk populations. To provide program
services, the ADE contracts with other organizations including
school districts, an adult education program, a community col-lege,
and community-based organizations.
Contractors currently provide services in five counties—
Maricopa, Pima, Cochise, Coconino, and Yuma. During the 1999-
2000 school year, 13 contractors operated 23 program sites, 12 of
which were in Maricopa County. Six contractors operated single
sites and seven contractors operated multiple sites. By statute,
each site must enroll at least 10 families but no more than 20
families. Table 1 (see page 7), shows the contractors and sites
operating in each county and the number of participants served
at each site. Some sites served more than 20 families over the
course of the school year, but only served between 10 and 20
families at one time. See the Appendix (pages a-i through a-xi)
for information on sites’ completion rates. Of the 391 families
served by Family Literacy contractors, 209 (53 percent) reside in
Maricopa County.
The Legislature continued to appropriate $1 million for the Fam-ily
Literacy Program for each fiscal year between 1996 and 2001.
ADE does not retain any funds for administration. All monies
appropriated for the Family Literacy Program are awarded to
contracted sites. Any leftover funds are nonreverting and can be
carried over to the next year.
All of the program sites have funding collaborators who provide
financial support in addition to the state grant. Resources re-quired
to operate a family literacy program vary depending on
the number of days per week the program operates, the number
of participants, and the cost of conducting the program in a par-ticular
community. The NCFL estimates that one full-time pro-gram
site requires $50,000 to $90,000 per year in a rural area and
Introduction and Background
7
OFFICE OF THE AUDITOR GENERAL
Table 1
Family Literacy Program
County, Contractor, Program Site, and Number of Families Served
1999-2000 School Year
(Unaudited)
County and Contractor Program Site Number of Families
Cochise
Cochise Community College La Escuelita 12
St. Paul’s 15
Coconino
Flagstaff USD No. 1 Leupp 11
W. F. Killip 20
Maricopa
Glendale ESD No. 40 Lamar Learning School 19
Isaac ESD No. 5 Isaac School, Site A 18
Isaac School, Site B 11
Literacy Volunteers of
Maricopa County C. J. Jorgensen 20
J. R. Davis 15
Littleton ESD No. 65 Littleton School 21
Mesa USD No. 4 Eisenhower 19
Lincoln 19
Longfellow 18
Phoenix Indian Center Longview 21
Southwest Human Development Umon Center 14
Tempe ESD No. 3 Frank School 14
Pima
Pima County Adult Education Liberty 25
Nash 16
Prince 17
South Tucson 11
Yuma
Crane ESD No. 13 H. L. Suverkrup, Site A 20
H. L. Suverkrup, Site B 20
Somerton ESD No. 11 Desert Sonora Elementary 15
Total participants 391
Source: Auditor General staff analysis of data provided by the Arizona Department of Education and Family Literacy
program staff.
Introduction and Background
8
OFFICE OF THE AUDITOR GENERAL
$75,000 to $125,000 in an urban area. As illustrated in Table 1 (see
page 7) and Table 2 (see page 9), for fiscal years 1999 through
2001, the contract amounts ranged from $30,100 to $65,050 for
contractors with single classroom sites and $59,700 to $226,500
for contractors with multiple sites. By dividing the total 1999 to
2000 state contract amounts by the number of families served,
Auditor General staff calculated the average cost to the state per
family to be $2,533. The long-term savings associated with the
program will be assessed in the next annual evaluation when
school data regarding student performance from the last decade
will be available.
Scope and Methodology
This evaluation, like those of previous years, examines program
implementation and participant outcomes. It reports on how the
program is administered by ADE, model programs, and pro-gram
contractors. As in past reports, the evaluation also de-scribes
participant outcomes and compares them to the out-comes
found in other programs, as well as to outcomes reported
for the previous school year.
n Program administration—To assess the administration’s
organizational structure and effectiveness in meeting statu-tory
requirements, OAG staff conducted staff interviews, file
reviews, a survey of program providers, and structured site
reviews. During the 1999-2000 school year, all 23 program
sites received structured reviews. As part of these site visits,
program classroom sessions were observed, site documents
were reviewed, and program staff and participants were in-terviewed.
A survey of program providers was administered
to describe the staff and collaborative partnerships found at
each site.
n Participant outcomes—Methods used to assess participant
outcomes include (1) analyses of the employment and educa-tional
gains of adult participants as noted on exit forms and
Introduction and Background
9
OFFICE OF THE AUDITOR GENERAL
Table 2
Family Literacy Program
Schedule of Contractors and Contract Awards
Years Ended or Ending June 30, 1999, 2000, and 2001
(Unaudited)
Contractor 1999 2000 2001
Cochise County
Cochise Community College $ 89,200 $ 89,200 $ 89,200
Coconino County
Flagstaff USD No. 1 59,700 59,700 59,700
Maricopa County
Glendale ESD No. 40 40,000 a 40,000 40,000
Isaac ESD No. 5 71,400 65,050 a,b,c 71,400
Literacy Volunteers of Maricopa County 67,900 67,900 67,900
Littleton ESD No. 65 40,000 a 65,050 a,b,c 40,000
Mesa USD No. 4 168,100 a 169,000 a 168,100
Phoenix Indian Center 36,600 36,600 36,600
Southwest Human Development 30,100 30,800 a,c 30,100
Tempe ESD No. 3 38,200 a 39,350 c 38,200
Pima County
Pima County Adult Education 222,000 226,500 a 220,000
Yuma County
Crane ESD No. 13 70,300 a 70,300 a 70,299
Somerton ESD No. 11 46,500 a 46,500 a 46,500
Total $980,000 $1,036,650 $977,999
a These contractors did not entirely expend their contract award in fiscal year 1999 and 2000. Approximately $26,200
and $46,100 remained unspent at June 30, 1999 and 2000, respectively.
b These contractors received a total of $47,400 in additional allocations for Internet connectivity that were made after
the original contract.
c These contractors received a total of $3,150 in additional allocations for an adult education conference that were
made after the original contract.
Source: Auditor General staff analysis of data provided by the Arizona Department of Education.
(2) comparisons of scores at the end of the program to those
at the beginning of the school year on the following:
Ø Results of standardized tests designed to measure:
— adult basic education skills (Tests of Adult Basic Educa-tion),
Introduction and Background
10
OFFICE OF THE AUDITOR GENERAL
— English language skills (Language Assessment System
and Comprehensive Adult Student Assessment System),
and
— parenting skills (Parent as a Teacher Inventory);
Ø Adult’s parenting behavior scores (Behavior Frequencies
Assessment); and
Ø Children’s development scores (Pre-K Success).
Participants from the 1999-2000 school year took standardized
tests upon entering the program and at the end of the school
year. To most accurately measure the impact of the program,
which runs through the entire school year, this evaluation re-ports
on test data only for participants who were administered
all instruments at appropriate times throughout the school year.
To assess how these test scores compare with those of another
similar program, the results were compared to those of partici-pants
in the 1999-2000 Even Start Program. This federally funded
family literacy program also adheres to the model for family
literacy advanced by the NCFL.
Besides being compared to the Even Start Program, the pre-school
component of the Family Literacy Program was also
compared to the Early Childhood Block Grant program (ECBG).
The ECBG program includes a state-funded preschool program
targeting at-risk children. While the Even Start group provided a
comparison with a similar family literacy program, the ECBG
program provided a comparison with a stand-alone preschool
program, a preschool program without the adult and parenting
components. See the Appendix (pages a-i through a-xi) for more
information on comparison groups.
Summary of Evaluation
Findings and Comparisons
to Previous Outcome Findings
This year’s evaluation reports improvements in program ad-ministration
and site operation. Overall, participants made im-provements
comparable to those in other programs, but some
improvements were only modest. This year’s evaluation reports
findings in the following four areas:
Introduction and Background
11
OFFICE OF THE AUDITOR GENERAL
n Program administration—ADE improved its administra-tion
and oversight of the program and all sites operated in
accordance with statutory requirements and program model
guidelines (see Finding I, pages 13 through 19).
n Adult education and employment—Family Literacy adults
improved their English language skills, educational ad-vancement,
and entry into the workforce. While they made
no significant gains in most tests measuring adult basic edu-cation,
their scores were at least comparable to those made by
Even Start participants (see Finding II, pages 21 through 27).
n Parenting—Family Literacy adults made significant im-provements
in their parenting behaviors, but made lesser
gains in parenting attitudes than Even Start adults (see Find-ing
III, pages 29 through 32).
n Children’s development—Children in the Family Literacy
Program improved their kindergarten readiness skills and
these gains were comparable to those of the children in Even
Start and ECBG preschool comparison groups (see Finding
IV, pages 33 through 38).
Table 3 (see page 12), includes this report’s and previous reports’
findings on the comparisons of Family Literacy outcomes to
those of other programs.
Introduction and Background
12
OFFICE OF THE AUDITOR GENERAL
Table 3
Family Literacy Program
Compared to Even Start and Early Childhood Block Grant Programs
Participant Outcomes
1996-97, 1998-99, and 1999-00 School Years
Comparison Results
Outcomes
Comparison
Program 1996-97 1998-99 1999-00
Adult Education and Employment
English language skills Even Start Similar Similar Similar
GED attainment Even Start Similar Less than
Even Start
Similar
Adult basic education Even Start Similar Data not avail-able
Similar
Entry into workforce Even Start Similar Similar Similar
Parenting outcomes
Parenting attitudes Even Start Similar Less than
Even Start
Less than
Even Start
Parenting behavior Even Start Similar Similar Similar
Children’s development outcomes
Total kindergarten
readiness
Even Start
Early Childhood
Block Grant
Greater than
Even Start
Not compared
Not compared
Less than Block
Grant
Similar
Similar
Source: Auditor General staff analysis of data provided by the Arizona Department of Education and Family Literacy, Even
Start, and Early Childhood Block Grant program staff.
Acknowledgements
The Auditor General and staff thank the Superintendent of Pub-lic
Instruction; the staff of the Arizona Department of Educa-tion’s
Adult Education Division; staff from the University of
Arizona’s Department of Educational Psychology; the model
program administrators; and the staff and families of the Family
Literacy, Even Start, and ECBG programs for their cooperation
and assistance.
13
OFFICE OF THE AUDITOR GENERAL
FINDING I ADE IMPROVES PROGRAM
ADMINISTRATION AND SITES
FOLLOW STATUTORY AND
MODEL PROGRAM GUIDELINES
ADE’s Adult Education Division has improved its administra-tion
of the Family Literacy Program but needs to keep building
on these improvements. Last year’s evaluation highlighted the
need for better administration when it reported that sites were
not complying with basic requirements for the number of fami-lies
enrolled or the implementation of program components. In
response, ADE hired a coordinator to monitor the program, co-ordinated
with model program sites to provide technical assis-tance
to other sites, and began developing better systems for
gathering information from the sites. This year’s evaluation re-ports
that all sites are in compliance, although some sites are not
administering tests or reporting results in a timely or accurate
manner.
ADE Improved Program
Oversight and Administration
Last year’s evaluation reported that 7 of 23 sites were out of
compliance with at least one statutory requirement and/or pro-gram
goal. Since that time, ADE improved program administra-tion
and oversight through several measures. These include
hiring a coordinator to oversee the sites, coordinating with two
model program’s sites to provide training and guidance to other
sites, and developing systems to gather data about the sites’ ac-tivities.
Family Literacy Coordinator position developed—At the end of
the 1998-1999 school year, ADE created and filled an administra-tive
position to oversee the Family Literacy Program and the
federally funded Even Start family literacy program. Funded
through sources other than the program appropriation, the Fam-ily
Literacy Coordinator position improved ADE’s administra-tive
oversight of family literacy programs by:
Finding I
14
OFFICE OF THE AUDITOR GENERAL
n Visiting all sites to monitor compliance—To assess each
site’s compliance with statutory requirements and program
goals, ADE’s Family Literacy Coordinator conducted struc-tured
site visits to all 23 Family Literacy sites and provided
each with technical assistance reports based on the results of
the visit.
n Taking action to improve compliance—ADE required
proof of recruiting efforts from a site that was previously out
of compliance due to low enrollment numbers. ADE also re-quired
another site previously out of compliance to send
monthly attendance rosters to prove current enrollment
compliance.
n Communicating with staff at the sites—In addition to site
visits, ADE maintained contact with program staff through
an adult education conference and a mandatory meeting of
all program directors and administrators. The Family Liter-acy
Coordinator reviewed end-of-year reports submitted by
each site to further assess sites’ compliance and to recom-mend
program improvements.
Coordination with model programs improved—ADE continued
to contract with two model programs to provide other sites with
technical assistance and training in program goals. Model pro-grams
maintained regular communication with sites and visited
each family literacy site at least twice during the year. Through-out
the year, model programs provided a variety of training
sessions for program staff including an annual conference on the
NCFL family literacy model program components, multiple
training sessions in test administration, professional develop-ment
workshops, and individual site workshops.
ADE also improved its coordination with model program staff
regarding the guidelines and time frames for test administration.
When requested by sites, model program staff provided addi-tional
training in areas such as test administration and team
building.
Systems for reporting program information developed—ADE
also continued to develop ways to enhance the collection and
analysis of participant information and test score data from the
sites by:
ADE’s Family Literacy Coor-dinator
visited sites and moni-tored
compliance with statu-tory
requirements and pro-gram
goals.
Finding I
15
OFFICE OF THE AUDITOR GENERAL
n Contracting with a data specialist—to develop and main-tain
a database containing Even Start and Arizona Family
Literacy program information and provide a report on the ef-fectiveness
of Even Start and Arizona Family Literacy pro-grams.
n Developing two on-line reporting systems—one for adult
education information and one for parents’ and children’s
information. The adult education reporting system became
fully operational on July 1, 2000. The parents’ and children’s
test score information system will be piloted in the 2000-2001
school year.
All Sites Operate According to
Statutory Requirements
and Program Guidelines
During the 1999-2000 school year, all Family Literacy program
sites operated in compliance with statutory requirements and
program goals. The previous school year, sites had difficulty
meeting statutory requirements for enrollment, eligibility, and
program components. Sites that were previously out of compli-ance
are now in compliance, or are no longer operating.
n Enrollment and eligibility—During the 1998-1999 school
year, 4 sites had difficulty maintaining the required enroll-ment
of at least 10 (and no more than 20) families throughout
the year and 1 of those sites enrolled ineligible families. One
of the noncomplying sites was closed and replaced by a new
site. In 1999-2000, all sites met the enrollment requirements.
On average, each site served 14 families at a time throughout
the year.1 Many sites continued to recruit and enroll families
during the year to remain in compliance with enrollment cri-teria.
n Program components—The report covering the 1998-1999
school year found that four sites failed to sufficiently inte-
1 Sites were in compliance if they maintained 10-20 eligible families in their
first month of operation, or for a majority of the months that they were op-erating.
In the 1999-2000 school year, sites enrolled between 5 to 22 families
month-to-month.
Finding I
16
OFFICE OF THE AUDITOR GENERAL
grate the program model components. Two of them were
closed and replaced by new sites.1 This year, all sites were
found in compliance with program guidelines because they
demonstrated the presence and integration of all four pro-gram
components: adult education, early childhood educa-tion,
Parent and Child Together time, and Parent Time.
Continued Administrative
Oversight Needed to
Ensure Compliance
Previous OAG evaluations have found that site compliance with
statutory and programmatic requirements have fluctuated with
ADE’s intensity of administrative oversight. In years when ADE
improved administrative oversight of the program, compliance
improved. In years when oversight was weak, compliance di-minished.
Table 4 (see page 17), shows how compliance and
oversight have varied in previous years.
Further Improvements
Needed at Site Level
While ADE’s efforts show promise, producing more complete
and accurate information for program evaluation, further im-provements
are needed at the site level. While all sites were in
compliance with statutory and program guidelines, 10 of the 23
sites did not follow contractual guidelines for administering tests
and reporting results. A change in the reporting requirements
would help identify those sites that are not administering tests
and reporting data in a timely manner.
1 After the 1998-1999 school year, two sites were closed by the provider
because they were not operating well. One site failed to meet the program
model requirements and the other failed to meet either enrollment or pro-gram
model requirements. The contractor that operated both sites replaced
them with new sites for the 1999-2000 school year.
While all sites operated
incompliance with statutory
requirements, some sites did
not report complete and
accurate test score data.
Finding I
17
OFFICE OF THE AUDITOR GENERAL
Table 4
Family Literacy Program
ADE Oversight and Site Compliance
1996 through 2000 School Years
School
year
Description of
ADE oversight
Were most sites
in compliance?
1995-1996 ADE staff visited only a few sites and lacked
a structured process to review the sites they
did visit.
No. Only 7 of 26 sites met all statutory and
program requirements.
1996-1997 ADE made structured visits to all sites, then
closely monitored the sites that had diffi-culty
with program guidelines.
Yes. 20 of 23 sites met all statutory and program
requirements.
1997-1998 ADE did not visit sites or collect complete
program data from sites.
Insufficient data to report on compliance or
program outcomes.
1998-1999 ADE made only five structured site visits
and did not collect sufficient data from those
and unvisited sites to make a complete
assessment of program outcomes.
Generally, yes, however, only 16 of 23 sites met
all statutory and program requirements.
1999-2000 ADE made structured site visits to all sites,
monitored sites that were previously out of
compliance, and developed a data collection
system so they could assess program out-comes.
Yes. All sites met statutory and program re-quirements.
Source: Auditor General staff analysis of data provided by Arizona Department of Education and Family Literacy Program
staff.
Ten sites did not meet testing and reporting requirements—ADE
requires contractors to submit the data needed to evaluate the
Family Literacy Program and provides contractors with guide-lines
for administering the tests, and deadlines for reporting.
While most contractors followed ADE’s contractual guidelines
for testing and data reporting, ten sites failed to report test scores
and/or send complete and accurate test data. See Appendix
(pages a-i through a-xi) for sites’ reporting compliance as well as
site rankings for each test.
n No pre- and posttest data reported—Four sites failed to
report children’s assessment pre- and/or posttest data. Three
sites failed to report data on tests measuring either adult ba-sic
education or English language skills.
Finding I
18
OFFICE OF THE AUDITOR GENERAL
n Inaccurate or incomplete data sent—One site sent inaccu-rate
data on the adult basic education test scores. Five sites
sent inaccurate or incomplete data on the children’s assess-ment
test scores. Some sites had difficulty using a test report-ing
software program. Other sites sent incomplete data and
left out information such as test dates and the age of the
child. As a result, some data could not be used for this year’s
evaluation.
Separate pre- and posttest reporting would help ensure data
completeness and accuracy—ADE required sites to report chil-dren’s
pretest and posttest scores together at the end of the year.
This time frame for reporting children’s pretest scores prevented
early detection of sites’ difficulties using the test software and
their failure to comply with ADE test administration guidelines.
Therefore, ADE should revise the timelines for data submission
so that sites submit all pretest and intake data by mid-school year
and exit and posttest data by the end of the school year.
Finding I
19
OFFICE OF THE AUDITOR GENERAL
Recommendations
1. ADE should continue monitoring activities to ensure that all
sites remain in compliance with statutory requirements and
program guidelines. In particular, ADE should continue an-nual
site visits and regular contact with all sites to determine
site compliance.
2. ADE should develop a policy to monitor contractors’ com-pliance
with contractual guidelines of data reporting. To do
so, ADE should:
a. Continue its efforts to develop a system for the collection
and entry of data and monitor the accuracy and com-pleteness
of all data submitted throughout the year,
b. Revise data submission timelines so that pretest and in-take
data are submitted by mid-school year and posttest
and exit data are submitted by the end of the school year,
and
c. Enforce data submission deadlines.
20
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
21
OFFICE OF THE AUDITOR GENERAL
FINDING II ADULT PARTICIPANTS MAKE
GAINS IN LANGUAGE PROFICIENCY
AND EMPLOYMENT OUTCOMES
During the 1999-2000 school year, Family Literacy program adult
participants made gains in developing their education skills and
joining the workforce. In both areas, they made gains that were
comparable to those made by participants in the federally
funded Even Start program. While participants made improve-ments
in their language skills, they did not show improvement
in most tests measuring adult basic education.
Program Provides Two
Educational Tracks: English
Proficiency and Adult Basic Education
When adult participants enter the Family Literacy Program, they
enroll in one or both of two educational tracks, depending on
their goals and language skills. The English for Speakers of Other
Languages (ESOL) track emphasizes English language devel-opment.
The Adult Basic Education (ABE) track advances stu-dents
through basic education topics, including GED prepara-tion
for advanced students. Adults who speak English enroll in
an ABE class, which may be offered in beginning (grades 0-4),
intermediate (grades 5-8), or secondary (grades 9-12) levels, de-pending
on the site. Adults who do not speak English enroll in
an ESOL class, and those who speak some English may enroll in
both ESOL and ABE classes. Of the 293 adult participants who
completed exit forms, 181 indicated that they participated in an
ESOL class, while 74 indicated that they participated in an
ABE/GED class and 38 participated in both.
Participants Improve
English Proficiency
Family Literacy program adult participants who took ESOL
classes in the 1999-2000 school year made statistically significant
Finding II
22
OFFICE OF THE AUDITOR GENERAL
gains in reading, writing, and listening comprehension skills.
Their gains were greater than those of Even Start program par-ticipants
in reading and comparable to those of Even Start par-ticipants
in writing, combined reading and writing, and listening
comprehension.
Gains were greater in reading, comparable in writing—Reading
and writing skills are measured by the Language Assessment
System (LAS). Family Literacy ESOL students made statistically
significant gains in both skills. The LAS includes a reading, writ-ing,
and combined reading/writing test. Reading and writing
scores are grouped into five categories of competence:
n Low beginner (0-20);
n High beginner (21-40);
n Low intermediate (41-60);
n High intermediate (61-80); and
n Competent (81-100)
Figure 4
Family Literacy Program
Participant Enrollment in Adult Education Classes
1999-2000 School Year
0%
20%
40%
60%
80%
100%
Note: Of the 391 families served by the Family Literacy Program, 293 provided information for this question.
Source: Auditor General staff analysis of data provided by the Arizona Department of Education and Family
Literacy Program staff.
Adult Basic Education (74)
Dual Enrollment (38)
English Speakers of
Other Languages (181)
Finding II
23
OFFICE OF THE AUDITOR GENERAL
The “competent” category was developed with the assumption
that “with few exceptions, literate adults from English-speaking
backgrounds should be able to pass the [subtests] as ‘Compe-tent.’
1 As seen in Figure 5, the reading scores for all adults ad-vanced
20 points, on average, to the “high intermediate” level
and the writing scores increased approximately 20 points, on
average, to the “low intermediate” level. About one-third of the
130 Family Literacy program adult participants who took pre-and
posttests in reading advanced to the “competent” level.
1 This guideline is provided in the LAS manual (DeAvila, Edward A. and
Sharon E. Duncan. Adult Language Assessment Scales Administration and
Scoring Manual. Monterey, CA: CTB-MacMillan/McGraw-Hill, 1993.)
Figure 5
Family Literacy Program
Improvement in Language Assessment System Test Scores
1999-2000 School Year
51.4
71.4
31.5
50.7
0
10
20
30
40
50
60
70
80
90
100
Reading Writing
Pretest Posttest
a The pretest to posttest improvement was significant at the .01 level. That is, the probability that the improve-ment
occurred by chance is less than 1 in 100.
Source: Auditor General staff analysis of data provided by Arizona Department of Education and Family Liter-acy
Program staff.
Competent
(81 to 100)
High Intermediate
(61 to 80)
Low Intermediate
(41 to 60)
High Beginner
(21 to 40)
Low Beginner
(0 to 20)
Improvement = 20.0 a
Improvement = 19.2 a
Finding II
24
OFFICE OF THE AUDITOR GENERAL
Although Family Literacy program participants had significantly
greater gains than Even Start program participants on the read-ing
test, the two groups had comparable gains on the writing
and combined reading and writing portions of the LAS. 1
Gains also comparable in listening comprehension—Listening
comprehension skills are measured by the Comprehensive Adult
Student Assessment System (CASAS). Family Literacy ESOL
students made statistically significant gains in their listening
comprehension skills.2 They increased their pretest scores on
average by 9.6 points to 211 points on a scale ranging from 0 to
241. This advanced their scores from a level at which students
could “answer basic questions” and “follow simple directions”
to a level at which they could “begin and end interview appro-priately”
and “can respond to multiple-step oral instructions.”
The 9.6 points average gain in scores for Family Literacy adults
was slightly higher than the 7.6 points gain for Even Start ESOL
participants, but the difference was not statistically significant.
Participants Show Gains in
Educational Attainment, But Not
in Tests of Adult Basic Education
During the 1999-2000 school year, adults who participated in the
Adult Basic Education track of the Family Literacy Program, on
average, did not improve in their scores on most adult basic
education test areas. However, neither did participants in the
Even Start program. Participants in both programs made pro-gress
in other measures of educational attainment, such as pass-ing
the GED.
No significant improvement in most Adult Basic Education test
areas—Education skills are measured by the Test of Adult Basic
Education (TABE), which includes tests on math, language,
1 The difference between Family Literacy and Even Start gains was signifi-cant
at the .05 level. That is, the probability that the difference occurred by
chance is less than 5 in 100. See the Appendix (pages a-i through a-xi) for
the average scores for each program.
2 The gains were statistically significant at the .01 level. See the Appendix
(pages a-i through a-xi) for the average scores of each program.
Family Literacy ESOL
students improved their
reading, writing, and listen-ing
comprehension skills.
Finding II
25
OFFICE OF THE AUDITOR GENERAL
spelling, and reading. During the 1999-2000 school year, 35 of the
112 Family Literacy ABE students took pre- and posttests in all
four tests. The average pretest to posttest gains ranged from 3 to
23 points on a 0-999 scale, and only the language test gain (23
points) was statistically significant. These small average pretest
to posttest differences were comparable to those made by Even
Start program participants. See the Appendix (pages a-i through
a-xi) for further discussion and the average TABE scores for both
programs.
Participants’ progress seen in other measures of educational
attainment—Eleven adults who participated in the Family Liter-acy
program during the 1999-2000 school year were accepted to
or had attended a community college, college, or university. Of
the 30 participants who indicated that they took the GED in that
school year, 10 passed all sections and 20 did not complete or
pass all sections yet. This shows improvement over the previous
school year when only four students reported that they received
their GED or high school diploma. Of the 21 Even Start program
adults who took the GED, 11 passed all sections.
During the 1999-2000 school year, Family Literacy participants
also reported gains in several other related programs or activi-ties.
n 19 Family Literacy program adults obtained their Child De-velopment
Associate credentials, which is a minimum re-quirement
for staff positions at most preschool programs.
n 121 Family Literacy program adults obtained their First Aid
card and/or CPR certification.
n 205 Family Literacy program adults obtained library cards
for their families.
n 7 Family Literacy program adults obtained U.S. citizenship
while enrolled in the program.
Finding II
26
OFFICE OF THE AUDITOR GENERAL
Employment Outcomes
Similar for Both Programs
Family Literacy and Even Start program adult participants re-ported
similar gains in employment during the 1999-2000 school
year.
Comparable entry into the workforce—Forty-two (23 percent) of
the 186 Family Literacy program adult participants who were
unemployed at the start of the program reported that they were
working by the end of the school year. Similarly, 55 (25 percent)
of the 216 Even Start program adults who were unemployed
upon entering the program reported that they were employed
by the end of the school year.
Employment results similar to comparison group and previous
year’s results—Approximately one-third (34 percent) of the 256
Family Literacy program adults who completed exit forms were
employed at the end of the program. Approximately another
third of the participants were unemployed and not looking for
work, and the remainder were also unemployed but either look-ing
for work or enrolled in a job training program. As seen in
Figure 6 (see page 27), these results were similar to those of Even
Start program participants as well as the results found in the
1998-1999 school year.
Employment gains were
comparable to those of Even
Start program adults and last
year’s participants.
Finding II
27
OFFICE OF THE AUDITOR GENERAL
Figure 6
Family Literacy Program
Compared to Even Start Program
Participants’ Employment Status at Exit
1999-2000 and 1998-99 School Years
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Percentage of Participants
Number of respondents was 256. Number of respondents was 271.
n=92
n=26
n=50
n=88
n=122
n=22
n=37
n=90
1998-1999 School Year
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Percentage of Participants
Number of respondents was 308. Number of respondents was 235.
Employed In job training program (unemployed)
Looking for work (unemployed) Not looking for work (unemployed)
Source: Auditor General staff analysis of data provided by Arizona Department of Education and Family Literacy and
Even Start program staff.
n=139
n=30
n=35
n=106
n=110
n=22
n=26
n=77
Family Literacy Even Start
Even Start Family Literacy
1999-2000 School Year
28
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
29
OFFICE OF THE AUDITOR GENERAL
FINDING III ADULT PARTICIPANTS SHOW
IMPROVEMENTS IN PARENTING
ATTITUDES AND BEHAVIORS
Adults enrolled in the parenting skills component of the Family
Literacy Program improved both their attitudes about parenting
and their parenting behaviors. Compared to participants in the
federally funded Even Start program, their gains in parenting
attitudes were less and their gains in actual parenting behaviors
were similar. These results are consistent with the evaluation for
the 1998-1999 school year (see Auditor General Report No. 00-03,
issued in March 2000).
Program Seeks to Improve
Both Attitudes and Behaviors
The Family Literacy Program focuses on developing parenting
skills through two components: Parent and Child Together
(PACT) time, and Parent Time group discussion. PACT time
provides an opportunity for parents to play with their children in
child-led and structured activities. With assistance and feedback
from program staff, parents learn to practice supportive parent-ing
behaviors. During Parent Time, adults discuss parenting and
early childhood development topics.
To assess the impact of the Family Literacy Program on parent-ing
attitudes and behaviors, adult participants took pre- and
posttests on the following two instruments:
n Attitudes—Parenting attitudes were measured by using the
Parent as a Teacher Inventory (PAAT). The PAAT assesses
parents’ attitudes in five areas: parental frustration, need for
control, understanding of play, acceptance of a child’s crea-tivity,
and views about their ability to help their child’s de-velopment.
n Behaviors—The adults’ parenting behaviors were measured
by the Behavior Frequencies Assessment (BFA). Five indica-
Finding III
30
OFFICE OF THE AUDITOR GENERAL
tors in the BFA measure behavior on a weekly basis, such as
the number of times they read to their children. Another five
indicators measure behavior on a monthly basis and focus on
school-related activities, such as helping children with their
homework.
Parenting Attitudes Improve,
But Not as Much as Those of
Even Start Participants
Most Family Literacy program parents entered the program with
positive parenting attitudes that were strengthened further in the
program. Although parenting attitudes improved for partici-pants
in both programs, adults in the Even Start program made
significant improvement over those in the Family Literacy Pro-gram.
This was also the case in the evaluation for the 1998-1999
school year (see Auditor General Report No. 00-03, issued in
March 2000).
Most parents show modest gains in already positive parenting
attitudes—Adults entered the program with positive parenting
attitudes overall. During the 1999-2000 school year, adults who
took PAAT pre- and posttests had an average pretest score of
139, which is already above the 125-point cut-off for “favorable”
parenting attitudes on the 0-200 scale. On average, adult pro-gram
participants improved their scores by 4 points, to an aver-age
posttest score of 143 points.1 This outcome was similar to
results reported in the evaluation for the 1998-1999 school year.
See Appendix (see pages a-i through a-xi) for specific results.
Average gains greater for Even Start participants—Even Start
program participants increased their PAAT scores by 9 points on
average, more than twice as much as Family Literacy program
participants’ average gains.2 The previous evaluation found
similar results for the 1998-1999 school year.
1 There was statistically significant improvement in all of their subscores
except for the subtest on “parental understanding of play.”
2 The difference between the two programs’ total gains was statistically
significant at the .01 level. That is, the probability that the difference oc-curred
by chance is less than 1 in 100.
Most Family Literacy pro-gram
adults enter with posi-tive
parenting attitudes.
Finding III
31
OFFICE OF THE AUDITOR GENERAL
We examined but could not find an explanation for the Even
Start adults’ significantly greater increases in PAAT scores. For
example, Even Start adults could have had greater familiarity
with the PAAT assessment because they can spend more years
in the program compared to Family Literacy program adults.
However, when only first-year participants were compared,
Even Start participants still showed significantly higher gains
than Family Literacy adults.1 We also compared the pretest aver-ages
for the two groups because lower pretest scores can be asso-ciated
with greater gains. However, the groups’ pretest scores
were similar. Additionally, differences between program models
could have explained different gains, but the model was the
same for both programs. Other factors, such as time in the pro-gram
and hours of instruction, also failed to explain the differ-ences
in improvement.
Adults Increase Positive
Parenting Behaviors
As measured by the BFA, Family Literacy program adults re-ported
significant increases in the number of times they demon-strated
positive parenting behaviors. Even Start program adults
showed similar gains in positive parenting behaviors from pre-test
to posttest.
n Family Literacy—This program’s adults increased positive
parenting behaviors at home and in activities associated with
their children’s school. On average, they improved their
scores on all 10 indicators on the BFA.2 As seen in Table 5
(see page 32), upon entering the program, Family Literacy
program adults took their children to the library less than
once a week (on average, 0.8 times per week). By the end of
the program, they reported that they went to the library with
their children at least twice as often (on average, 1.6 times per
week). Also, at the end of the program, they almost doubled
the number of times they participated in activities associated
with their children’s school.
1 The difference between Family Literacy and Even Start first-year partici-pants’
average gains was statistically significant at the .01 level.
2 These improvements were statistically significant at the .01 level.
Similar to last year’s results,
the Even Start program had a
greater impact on parenting
attitudes than the Family
Literacy Program.
Finding III
32
OFFICE OF THE AUDITOR GENERAL
n Even Start—This program’s adults also reported improve-ments
in their parenting behaviors. These adults showed sig-nificant
improvements on 8 of 10 BFA indicators. Even Start
adults’ improvements over their pretest scores were compa-rable
to those of Family Literacy participants on all but the
number of times they talked to their children about school.
However, on that indicator, Even Start adults’ average pre-test
score was as high as the Family Literacy average posttest
score. While Family Literacy adults improved by talking to
their children about school six more days per month, there
was less room for improvement for the Even Start adults in
that behavior.
Table 5
Family Literacy Program
Compared to Even Start Program
Changes in Frequency of Parenting Behaviorsa
1999-2000 School Year
Parenting Behavior
Family Literacy
Number of Days
Even Start
Number of Days
Start End Change Start End Change
Number of days per week parents:
Talk about children’s day 5.2 5.7 +0.5 5.4 5.6 +0.2
Read book with children 4.1 4.7 +0.6 4.0 4.8 +0.8
Let children see them writing 4.5 5.1 +0.6 4.8 5.1 +0.3
Take children to the library 0.8 1.6 +0.8 0.7 1.4 +0.7
Play with children 5.3 5.9 +0.6 5.3 5.7 +0.4
Number of days per month parents:
Go to a school activity 4.0 5.9 +1.9 3.5 6.7 +3.2
Volunteer for a school activity 2.2 4.3 +2.1 1.8 4.6 +2.8
Help children with homework 1.5 4.9 +3.4 2.2 5.4 +3.2
Talk to children’s teacher 8.7 14.4 +5.7 14.0 16.7 +2.7
Talk to children about school 10.0 16.0 +6.0b 16.2 18.2 +2.0b
a All gains from pretest to posttest were significant at the .05 level, except for Even Start participants’ average gain in the
number of days that they talked about their children’s day.
b The difference between Family Literacy and Even Start gains was significant at the .01 level. That is, the probability that the
difference occurred by chance is less than 1 in 100.
Source: Auditor General staff analysis of data provided by the Arizona Department of Education and Family Literacy and
Even Start program staff.
33
OFFICE OF THE AUDITOR GENERAL
FINDING IV CHILDREN IN FAMILY LITERACY
IMPROVE KINDERGARTEN
READINESS SKILLS
During the 1999-2000 school year, children in the Family Literacy
Program improved their readiness to succeed in kindergarten.
Analyses of the Family Literacy program children’s scores on a
kindergarten readiness assessment shows, that on average, they
increased their development by 7 to 10 months, which is compa-rable
to gains made by children in the Even Start and Early
Childhood Block Grant (ECBG) programs. While only a portion
of the Family Literacy program’s 4-year-olds advanced to kin-dergarten
readiness levels in all test measures, these results were
not only consistent with the Even Start and ECBG groups, but
also with other research on similar at-risk populations. Several
factors associated with the program were related to children’s
levels of improvement. ADE should continue to monitor imple-mentation
of model program recommendations including those
factors associated with improved kindergarten readiness.
Program’s Preschool Education
Component Aims to Enhance
Kindergarten Readiness
The Family Literacy Program strives to improve the school
readiness of the preschool children in the program by providing
developmentally appropriate preschool education. To measure
the program’s impact on the children’s kindergarten readiness,
children were administered pre- and posttests of the Pre-K Suc-cess
assessment (PKS). The PKS uses a developmentally appro-priate
testing methodology and provides a kindergarten readi-ness
assessment containing the following three subtests:
n Language—measures stages of vocabulary, sentence struc-ture,
expression, and comprehension.
Finding IV
34
OFFICE OF THE AUDITOR GENERAL
n Problem Solving—measures cognitive skills, such as the
classification and positioning of objects and the understand-ing
of quantities and characteristics of numbers; and
n Motor Skills—measures gross and fine motor skills, such as
balance, coordination, grasp, release, and the ability to sus-tain
a motor pattern.
The 1997 evaluation compared the Family Literacy program
children’s assessment scores to those of children in the federally
funded family literacy Even Start program. The evaluation for
1998-1999 compared Family Literacy children’s assessment
scores to those from the state-funded, stand-alone ECBG pre-school
program. This year’s evaluation compared the Family
Literacy children’s assessment scores to those of both the Even
Start and the ECBG programs.
Family Literacy Improvements
Were Similar to Comparison
Groups’ Improvements
Family Literacy program children made significant improve-ments
in the language, problem solving, and motor skills sub-tests,
with their greatest gains made in language skills. Children
in Family Literacy made similar gains to children in Even Start
and ECBG programs in all three subtests. Figure 7 (see page 35),
shows the Family Literacy program children’s average devel-opmental
gains reported in age equivalent months for each of the
subtests. See the Appendix (pages a-i through a-xi) for additional
test score information on all three programs.
n Language skills development—Language skill levels for
children in the Family Literacy Program increased an aver-age
of 10 months, from a pretest at the 3-year, 6-month level
to a posttest at the 4-year, 4-month level.1 This 10-month in-crease
is notable because the children participated in the pro-gram
for only 7 months, on average, between pretest and
posttest. ECBG program children improved by 9 months and
Even Start program children improved by 11 months.
1 The gain was statistically significant at the .01 level. That is, the probability
that the difference occurred by chance is less than 1 in 100.
Family Literacy program
children improved their lan-guage
skills by 10 months, on
average.
Finding IV
35
OFFICE OF THE AUDITOR GENERAL
Problem-solving skills development—Problem-solving skill
levels for children in the Family Literacy Program increased an
average of 9 months, from a 3-year, 9-month level to a 4-year, 6-
month level.1 This gain was comparable to those of Even Start
program children, who also began at the 3-year, 9-month level
and advanced by 10 months. ECBG program children had
slightly lower gains, increasing their problem-solving skills de-velopment
by an average of 7 months. This result was in contrast
to the evaluation for 1998-1999, which found that ECBG program
children had greater improvement in problem-solving skills than
Family Literacy program children.
1 The gain was statistically significant at the .01 level.
Figure 7
Family Literacy Program
Average Improvement in Developmental Skills Test Scores
Converted to Months
1999-2000 School Year
0
10
20
30
40
50
60
Average Scores Converted to Months
Language Motor Problem Solving
pretest posttest
Source: Auditor General staff analysis of data provided by the Arizona Department of Education and Fam-ily
Literacy Program staff.
Finding IV
36
OFFICE OF THE AUDITOR GENERAL
n Motor Skills Development—Family Literacy program chil-dren
developed their motor skills by 7 months, on average,
from a 4-year level to a 4-year, 7-month level. This statistically
significant improvement was comparable to the 6-month
gain made by Even Start program children and the 5-month
gain made by ECBG program children. Although Family Lit-eracy’s
program gain was similar to the ECBG program gain,
these results differ from last year’s report which found that
ECBG program children, on average, showed greater gains
than Family Literacy program children in motor skill devel-opment.
The modest improvements in motor skill develop-ment
by children in all three programs may be due to the
high pretest scores in this area. As children approach the
“ceiling” of a test scale, large gains are less likely.
Not All Program 4-Year-Olds
Advance to Kindergarten
Readiness Levels
The Family Literacy Program strives to bring all 4-year-olds in
the program to a level of kindergarten readiness in all 3 subtests,
but most do not reach that point. However, the program’s 4-
year-old gains are comparable to other at-risk programs.
Most 4-year-olds reach kindergarten readiness levels in at least
one subtest—Thirty-two (67 percent) of the 48 Family Literacy
program 4-year-olds who took pre- and posttests reached kin-dergarten
readiness levels in at least one of the three subtests.
More specifically, 15 (31 percent) of the 48 four-year-olds post-tested
at the kindergarten readiness levels in all three subtests,
six (13 percent) reached the kindergarten level in two of the three
subtests, and 11 (23 percent) reached that level in one of the three
subtests. The remaining 16 four-year-olds scored below kinder-garten
readiness levels in all three of the subtests.
Results conform to comparison groups and research on at-risk
populations—During the 1999-2000 school year, Even Start and
ECBG program 4-year-olds reached kindergarten readiness lev-els
in comparable proportions to the Family Literacy program
children (see Appendix, pages a-i through a-xi). The modest
kindergarten readiness results conform to research on at-risk
populations reported in the 1997 Auditor General evaluation of
Less than one-third of the
Family Literacy Program 4-
year-olds met kindergarten
readiness levels in all test
areas.
Finding IV
37
OFFICE OF THE AUDITOR GENERAL
the At-Risk Preschool Expansion Program (see Auditor General
Report No. 97-2). That evaluation found that at-risk children
tended to score below national norms on standardized assess-ments.
Further, educational experts who developed the Pre-K
Success caution against comparing the test scores of children in
lower income families with those from children in higher income
groups.
Program Can Build on Factors
Related to Children’s
Developmental Gains
For all three programs, children’s developmental gains tend to
increase under certain conditions. Three factors are associated
with higher gains: the number of home visits, the number of trips
to the library, and the parent’s participation in additional parent-ing
classes. The Family Literacy Program should take steps to
take full advantage of these factors in increasing children’s pre-paredness
for school.
n Home visits—Home visits by program staff were also re-lated
to children’s development. Although the NCFL family
literacy program model recommends but does not require
home visits, model program sites have increased the number
of times their staff members visit participants’ homes. Statis-tical
tests on the 1999-2000 school year data found that every
additional home visit was associated with another month
gain in children’s problem-solving skill development.
n Trips to the library—Going to the library is one of many
positive parenting behaviors and has obvious relevance for
improving family members’ literacy. When parents took their
child to the library one more day a week on average, the
children’s language skills increased by an additional month.
n Additional parenting classes—Some sites offer parenting
classes, such as Parent University, in addition to the model’s
parent education component. The parents’ participation in
these additional parenting classes correlated with an addi-tional
6-month increase in the children’s language skills.
Increases in home visits by
program staff, trips to the
library, and additional
parenting classes were
associated with greater
gains in children’s kinder-garten
readiness.
Finding IV
38
OFFICE OF THE AUDITOR GENERAL
Recommendation
ADE should continue to monitor implementation of model pro-gram
recommendations that are associated with improved kin-dergarten
readiness; specifically, more home visits to families,
improved positive parenting behaviors (such as going to the
library), and additional parenting classes for adults.
OFFICE OF THE AUDITOR GENERAL
APPENDIX
a-i
OFFICE OF THE AUDITOR GENERAL
APPENDIX
Introduction and Background
(See pages 1 through 12)
Site Completion Rates and Reasons for Exit
Table 6 (see page a-ii), shows the number of participants served
and the completion rates for each site. Of the 391 participants
served, 248 (63 percent) completed all portions of the program.
Site completion rates ranged from 35 to 94 percent. Of the 133
adults who disenrolled their families before completing the
program, 57 (43 percent) reported that they met their educational
goals. Twenty-nine (22 percent) reported that they left because
they had found employment that precluded their continued
participation. Seventeen (13 percent) moved out the area or to a
different program and 17 left because of a family crisis, conflict,
or personal reason. Only 13 (9 percent) dropped out of the pro-gram
because of a lack of interest, incomplete participation, or
poor attendance.
Comparison Groups
As with previous evaluations, this evaluation compared Family
Literacy participants’ outcomes to those of Even Start and ECBG
program participants.
n Even Start—Even Start is a federally-funded family literacy
program which implements the same four-component model
as the Arizona Family Literacy Program. Even Start partici-pants
have similar eligibility requirements as Family Literacy
participants, except Even Start families may enter the pro-gram
with children whose ages range from infancy to 7 years
old while Family Literacy children must be either 3 or 4 years
old. Even Start adults could enroll for longer periods of time
because of the extended eligibility of their children.
Appendix
a-ii
OFFICE OF THE AUDITOR GENERAL
Table 6
Family Literacy Program
Number and Percentage of Families Completing Program by Site
1999-2000 School Year
(Unaudited)
Contractor and Number
Program Site Served Completed
Percentage
Completed
Cochise Community College
La Escuelita 12 9 75%
St. Paul’s 15 8 53
Flagstaff USD No. 1
Leupp 11 8 73
W. F. Killip 20 7 35
Glendale ESD No. 40
Lamar Learning School 19 11 58
Isaac ESD No. 5
Isaac School, Site A 18 17 94
Isaac School, Site B 11 8 73
Literary Volunteers of Maricopa County
C. J. Jorgensen 20 17 85
J. R. Davis 15 7 47
Littleton ESD No. 65
Littleton School 21 8 38
Mesa USD No. 4
Eisenhower 19 13 68
Lincoln 19 11 58
Longfellow 18 14 78
Phoenix Indian Center
Longview 21 10 48
Southwest Human Development
Umon Center 14 12 86
Tempe ESD No. 3
Frank School 14 10 71
Pima County Adult Education
Liberty 25 15 60
Nash 16 7 44
Prince 17 9 53
South Tucson 11 7 64
Crane ESD No. 13
H. L. Suverkrup, Site A 20 13 65
H. L. Suverkrup, Site B 20 13 65
Somerton ESD No. 11
Desert Sonora Elementary 15 14 93
Total Number served 391 248
Average completion rate 63%
Source: Auditor General staff analysis of data provided by the Arizona Department of Education and Family Literacy
Program staff.
Appendix
a-iii
OFFICE OF THE AUDITOR GENERAL
n ECBG—The ECBG program is a state-funded program that
includes preschool tuition funding for at-risk children. There
is no curriculum or model for the preschools that receive
ECBG funds.
n Adult comparison group—The adult comparison group for
the last three evaluations has been Even Start participants.
Evaluations compared Family Literacy and Even Start pro-gram
adults on adult education test scores, education and
employment outcomes, and parenting test scores. During the
1999-2000 school year, the two programs served adult popu-lations
that were similar in terms of their gender, income,
employment status, language spoken, and education level.
Although the majority of adults in both programs were en-rolled
for the first time, Even Start had a slightly higher num-ber
of participants in their second and third years of the pro-gram.
n Child comparison groups—Previous evaluations have
compared Family Literacy children’s test scores to those of
either Even Start or ECBG program children. Children in
Even Start can be older, but during the 1999-2000 school year,
the children in Even Start were 4 years, 8 months old, on av-erage,
and Family Literacy children were 4 years, 4 months
old, on average. The ECBG children were 4 years, 5 months
old, on average.
This year’s evaluation compared Family Literacy children’s
scores to those of a sample of children in the ECBG program. The
sample size of the ECBG comparison group was 355 (10 percent
of the total 3,549 ECBG population and 13 percent of the 2,788
ECBG children in sites not receiving Family Literacy or Even
Start program funds). The sample size was chosen to match
approximately the sizes of the Even Start and Family Literacy
program participant groups. The sample was randomly chosen
from sites not receiving Family Literacy or Even Start program
monies and was stratified to reflect the same proportions of
participants in Maricopa County (56 percent) and those outside
of Maricopa County (44 percent) as found in the total ECBG
population.
Appendix
a-iv
OFFICE OF THE AUDITOR GENERAL
Finding I: ADE Improves
Program Administration and
Sites Follow Statutory and Model
Program Guidelines
(See pages 13 through 19)
While all sites followed statutory requirements and program
guidelines, some sites failed to comply with contractual obliga-tions
for reporting data and test scores to ADE. Table 7 (see page
a-v), shows which sites had difficulty reporting data and also
includes rankings of the sites’ average gains on all of the stan-dardized
assessments used in the report. When assessing sites’
success in test score gains or difficulty in reporting data, there
were no differences found among certain types of sites or con-tractors.
For example, sites that made the greatest gains on one
measure also made only modest gains on another.
Finding II: Adult Participants
Make Gains in Language
Proficiency and Employment Outcomes
(See pages 21 through 27)
n English Language Assessments—English language skills
of Family Literacy and Even Start program adults in ESOL
classes were measured by the Comprehensive Adult Student
Assessment System (CASAS) and the Language Assessment
System (LAS) tests. The CASAS scale ranged from 0 to 241,
the LAS reading and writing tests ranged from 0 to 100, and
the LAS combined reading and writing test ranged from 0 to
200. For each program and each assessment, Table 8 (see
page a-vi), reports the average pretest and posttest scores and
the average of individual gains between pretest and posttest.
Table 7
Family Literacy Program
Site Data Reporting Results and Rankings by Improvements on Test Scores
1999–2000 School Year
Tests of Adult Basic Education
Behavior
Frequencies
Type of Contractor Assessment
and Site
Comprehensive
Assessment
System of
Academic Skills
Language
Assessment
System Language Math Reading Spelling
Parent
As a
Teacher Weekly Monthly
Pre-K
Success
School District
Desert Sonora 14 12 1 7 6 7 5 15 7 10
Eisenhower a a 4 2 4 2 14 21 16 6
Frank 8 15 a a a a 19 18 10 5
H. L. Suverkrup, A 11 10 3 1 2 1 11 13 14 11
H. L. Suverkrup, B 12 11 a a a a 12 14 15 12
Isaac, A 2 9 9 9 7 6 10 3 4 I/D
Isaac, B 7 6 6 6 5 3 15 12 2 9
Killip 3 8 8 4 8 5 9 22 5 13
Lamar N/D 2 a a a a 13 20 8 14
Leupp a a I/D I/D I/D I/D 4 11 9 N/D
Lincoln a a 5 5 3 4 7 5 20 8
Littleton N/D N/D N/D N/D N/D N/D N/D N/D N/D N/D
Longfellow 16 13 a a a a 18 9 13 4
Community-based
C. J. Jorgensen 5 3 a a a a 2 4 12 I/D
J. R. Davis 1 4 a a a a 1 1 1 3
Longview 4 1 a a a a 21 2 1 N/D
Umon Center 18 19 a a a a 22 10 17 N/D
Adult education
Liberty 6 5 a a a a 3 8 19 7
Nash 15 18 a a a a 6 7 16 I/D
Prince 10 7 2 3 1 8 17 17 6 2
South Tucson 17 14 a a a a 20 16 21 1
Community College
La Escuelita 13 16 a a a a 16 19 3 I/D
St. Paul’s 9 17 7 8 8 N/D 8 6 11 I/D
a Test is not applicable to students; the site teaches only Adult Basic Education (ABE) or English for Speakers of Other Languages (ESOL) classes, but not both.
N/D No data is available because site did not administer tests or did not report pretest and posttest scores.
I/D Incomplete or inaccurate data reported by site.
Source: Auditor General staff analysis of data provided by the Arizona Department of Education and Family Literacy Program staff.
Appendix
OFFICE OF THE AUDITOR GENERAL a-v
Appendix
a-vi
OFFICE OF THE AUDITOR GENERAL
n Adult Basic Education Assessment—Changes in partici-pants’
levels of adult basic education were measured by the
Tests of Adult Basic Education (TABE), a battery of subject
area tests in language, math, reading, and spelling. Each sub-ject
area test is available in four levels: easy, medium, ad-vanced,
and difficult. Furthermore, each level contains a pre-test
and a posttest. The TABE scale ranges from 0-999. For
each program and each TABE test, Table 9 (see page a-vii),
reports the average pretest and posttest scores and the aver-age
of individual gains (and losses) between pretest and post-tests.
Although Even Start’s average pretest to posttest gains
were greater than Family Literacy’s average gains, the differ-ence
between the programs’ gains was not statistically sig-nificant.
Table 8
Family Literacy Program
Compared to Even Start Program
Average Improvements in English Language Assessmentsa
1999-2000 School Year
Assessment
Language Assessment System
Program Measures
Comprehensive
System of
Academic Skills Reading Writing Combined
Family Pretest 201.40 51.42 31.46 79.14
Literacy Posttest 211.03 71.42 50.69 119.21
Average of individual
gains 9.62
n=137
20.0b
n=139
19.23
n=125
40.07
n=137
Even Start Pretest 200.95 51.77 35.69 85.14
Posttest 208.51 66.66 54.88 119.76
Average of individual
gains 7.56
n=164
14.8b
n=177
19.19
n=162
34.6
n=176
a All pretest to posttest gains were significant at the .01 level.
b The difference between Family Literacy and Even Start gains was significant at the .05 level.
Note: “n” refers to the number of participants who completed pre- and posttests.
Source: Auditor General staff analysis of data provided by the Arizona Department of Education and Family Literacy and
Even Start program staff.
Appendix
a-vii
OFFICE OF THE AUDITOR GENERAL
Two factors may help explain the lack of improvement in Family
Literacy program participants’ average TABE scores. First,
students in the highest level ABE classes (equivalent to grades 9-
12) showed greater gains, on average, than the ABE students
attending beginning (grades 0-4) and intermediate (grades 5-8)
ABE classes. Approximately one-third of the ABE students who
took the TABE tests were in the beginning and intermediate ABE
levels and the large proportion of students in these levels may
have lowered the program’s average gains. Second, some stu-dents
took posttests at more challenging levels than their pre-tests.
If they were not prepared for the more difficult posttest
level, their posttests scores could be lower than their pretest
scores.
Table 9
Family Literacy Program
Compared to Even Start Program
Average Scores on Tests of Adult Basic Education
1999-2000 School Year
Program Measure Language Math Reading Spelling
Pretest 475.18 527.23 503.57 476.64
Posttest 498.55 530.40 507.94 493.64
Family Literacy
Average of individual gains 23.37a
n=51
3.18
n=40
4.37
n=51
17.00
n=47
Pretest 478.93 522.37 494.96 501.96
Posttest 522.63 542.81 518.41 515.81
Even Start
Average of individual gains 43.70b
n=27
20.44b
n=27
23.44b
n=27
13.85
n=27
a The pretest to posttest gain was significant at the .05 level.
b The pretest to posttest gain was significant at the .01 level.
Note: “n” refers to the number of participants who completed pre- and posttests.
Source: Auditor General staff analysis of data provided by Arizona Department of Education and Family Literacy and Even
Start program staff.
Appendix
a-viii
OFFICE OF THE AUDITOR GENERAL
Finding III: Adult Participants
Show Improvements in Parenting
Attitudes and Behaviors
(See pages 29 through 32)
Parenting Attitudes
Parenting attitudes were measured by using the Parent As a
Teacher Inventory (PAAT) which identifies favorable attitudes
and approaches needed for parenting children ages 3 to 9. In the
PAAT, adults respond to statements about their expectations of
and interactions with their child as well as the actions they
would take in response to specific behavior. The responses are
grouped into five areas with subscores computed for each area in
addition to the total score. The five areas are:
n Frustration—parental frustration with the child and focus of
the frustration;
n Control—parental feelings about the need to control the
child’s behavior;
n Play—parental understanding of play and its impact on
child development;
n Creativity—parental acceptance of the child’s creativity and
willingness to encourage its development; and
n Teaching/Learning—parental views about child develop-ment
and their ability to provide a supportive home environ-ment.
Finding III (see pages 29 through 32), discusses comparisons of
the total scores. The average total scores and the average of the
subtest scores for both programs are reported in Table 10 (see
page a-ix).
Appendix
a-ix
OFFICE OF THE AUDITOR GENERAL
Finding IV: Children in Family
Literacy Improve Kindergarten
Readiness Skills
(See pages 33 through 38)
For each child, the Pre-K Success (PKS) assessed kindergarten
readiness according to three subtests and calculated a composite
of those subtests into a total readiness score. Only the scales of
the three subtests are converted into age equivalencies by the test
publisher. The publisher does not convert the total readiness
scores to age equivalencies.
For each of the three programs, Table 11 (see page a-x), includes
the average of individual improvements in test scores converted
into age equivalencies.
Table 10
Family Literacy Program
Compared to Even Start Program
Average Improvements in Parenting Attitudes
1999-2000 School Year
Parenting Attitudes Toward
Program Measure Frustration Control Play Creativity Teach/Learn Total
Pretest 28.73 25.32 29.80 27.13 29.07 139.18
Posttest 29.28 25.95 30.29 28.08 30.16 142.82
Family
Literacy
Individual gains 0.55a,c
n=198
0.63a,c
n=198
0.50c
n=198
0.95b
n=197
1.09a,c
n=197
3.64 b,c
n=191
Pretest 28.40 25.29 29.32 26.72 28.73 138.42
Posttest 29.98 27.43 31.34 28.35 30.79 147.77
Even Start
Individual gains 1.58b,c
n=222
2.13b,c
n=222
2.02b,c
n=219
1.64b
n=222
2.06b,c
n=222
9.34b,c
n=219
a The pretest to posttest gain was statistically significant at the .05 level.
b The pretest to posttest gain was statistically significant at the .01 level.
c The difference between Family Literacy and Even Start gains was significant at the .01 level.
Note: “n” refers to the number of participants who completed pre- and posttests.
Source: Auditor General staff analysis of data provided by Arizona Department of Education and Family Literacy and Even
Start program staff.
Appendix
a-x
OFFICE OF THE AUDITOR GENERAL
Kindergarten Readiness Levels
Children were considered “kindergarten ready” if their posttest
scores were at the 5-year-old age equivalency level. For each of
the three subtests, the 4-year-olds’ scores were compared to the
5-year-old developmental skill level. Most posttests were given
between April 24 and May 17, 2000. Table 12 (see page a-xi)
includes the kindergarten readiness results for Family Literacy,
Even Start, and ECBG programs for the 1999-2000 school year.
Table 11
Family Literacy Program
Pre-K Success
Compared to Even Start and Early Childhood Block Grant Programs
Average Improvements in Pre-K Success Test Scores Converted to Months1
1999-2000 School Year
Developmental Skills
Family
Literacy
Even
Start
Early Childhood
Block Grant
Language 10 11 9
Motor 7 6 5
Problem solving 9 10 7
1 For each subtest and each program, the pretest to posttest gain was statistically significant at the .01 level. However, the
differences between Family Literacy and Even Start gains, and Family Literacy and Early Childhood Block Grant gains
were not statistically significant.
Source: Auditor General staff analysis of data provided by the Arizona Department of Education and Family Literacy,
Even Start, and Early Childhood Block Grant program staff.
Appendix
a-xi
OFFICE OF THE AUDITOR GENERAL
Table 12
Family Literacy Program
Compared to Even Start and Early Childhood Block Grant Programs
Number and Percentages of 4-Year-Olds Who Met Kindergarten Readiness Levels
1999-2000 School Year
Family Literacy Even Start
Early Childhood
Number of developmental Block Grant
skill levels met Number Percentage Number Percentage Number Percentage
All 3 15 31% 17 28% 45 22%
2 of 3 6 13 11 19 33 16
1 of 3 11 23 17 28 46 23
None 16 33 15 25 80 39
Source: Auditor General staff analysis of data provided by the Arizona Department of Education and Family Literacy,
Even Start, and Early Childhood Block Grant program staff.
a-xii
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
OFFICE OF THE AUDITOR GENERAL
Agency Response
1535 West Jefferson, Phoenix, Arizona 85007 l Voice: (602) 542-4391 l Fax: (602) 542-3050
State of Arizona
Department of Education
Lisa Graham Keegan
Superintendent of
Public Instruction
December 19, 2000
Ms. Debbie Davenport
Auditor General
State of Arizona
2910 North 44th Street Suite 410
Phoenix, AZ 85018
Dear Ms. Davenport:
I am responding to your agency's final evaluation of the Arizona Family Literacy Program.
A meeting to review the report draft was held, in our office, with members of my staff on
November 22, 2000. The discussion was very beneficial.
The report contains three recommendations for program improvement and our agency addresses
the recommendations as follows:
1. ADE should continue monitoring activities to ensure that all sites remain in
compliance with statutory requirements and program guidelines. In particular, ADE
should continue annual site visits and regular contact with all sites to determine site
compliance.
The finding of the Auditor General is agreed to and the evaluation recommendation will
be implemented. Currently, the state Family Literacy program is funded through a one-million
dollar annual appropriation that provides no funding for administration or
oversight. During the past year the ADE successfully began monitoring and oversight
activities through the use of Federal Adult Education State Leadership monies, however,
the federal government has reduced the state leadership allocation. Without state funding
to administer Family Literacy, continuation of monitoring and oversight activities will be
limited or in most cases discontinued.
2. ADE should develop a policy to monitor contractors’ compliance with contractual
guidelines of data reporting. To do this ADE should:
a. Continue its efforts to develop a system for collection and entry of data and
monitor the accuracy and completeness of all data submitted throughout the year.
1535 West Jefferson, Phoenix, Arizona 85007 l Voice: (602) 542-4391 l Fax: (602) 542-3050
b. Revise data submission timelines so that pretest and intake data are submitted
by mid-school-year and posttest and exit data are submitted by the end of the
school year and
c. Enforce data submission deadlines.
The finding of the Auditor General is agreed to and the evaluation recommendation will
be implemented. ADE has taken the following steps:
a. The data collection software development is progressing on schedule and is
anticipated to be completely implemented for FY2002.
b. ADE has changed the policy for submitting early childhood data to include a
mid-year collection of pretest scores. With the support of Family Literacy
program directors, ADE has changed the end of year data collection deadline
from June 30, 2001 to June 8, 2001. This should allow adequate time to carefully
review data for accuracy and completeness prior to the submission of the data to
the Auditor General’s office.
c. ADE will continue its enforcement of submission deadlines.
3. ADE should continue to monitor implementation of model program recommendations
that are associated with improved kindergarten readiness; specifically, more home visits
to families, improved positive parenting behaviors (such as going to the library) and
additional parenting classes for adults.
The finding of the Auditor General is agreed to and the audit recommendation will
continue to be implemented. ADE will require parental contacts in the FY2002 contracts
with its local providers. Linkage has been made with programs such as Parents
Anonymous to provide additional parenting classes for Family Literacy programs.
Programs are currently encouraged to assist parents in obtaining library cards and sites
often schedule field trips to local libraries. The ADE will continue to place emphasis on
activities that promote kindergarten readiness.
Our agency extends our appreciation to your staff for the high degree of cooperation and
assistance provided in the evaluation of the Arizona Family Literacy program.
Sincerely,
Lisa Graham Keegan
Superintendent of Public Instruction
Cc: Brian Jones
Jennifer Mabry
Karen Liersch
Lois Schneider
Other Performance Audit Reports Issued Within
the Last 12 Months
00-2 Behavioral Health Services—
Interagency Coordination of Services
00-3 Arizona’s Family Literacy Program
00-4 Family Builders Pilot Program
00-5 Arizona Department of Agriculture—
Licensing Functions
00-6 Board of Medical Student Loans
00-7 Department of Public Safety—
Aviation Section
00-8 Arizona Department of Agriculture—
Animal Disease, Ownership and
Welfare Protection Program
00-9 Arizona Naturopathic Physicians
Board of Medical Examiners
00-10 Arizona Department of Agriculture—
Food Safety and Quality Assurance
Program and Non-Food Product
Quality Assurance Program
00-11 Arizona Office of Tourism
00-12 Department of Public Safety—
Scientific Analysis Bureau
00-13 Arizona Department of Agriculture
Pest Exclusion and Management
Program
00-14 Arizona Department of Agriculture
State Agricultural Laboratory
00-15 Arizona Department of Agriculture—
Commodity Development Program
00-16 Arizona Department of Agriculture—
Pesticide Compliance and Worker
Safety Program
00-17 Arizona Department of Agriculture—
Sunset Factors
00-18 Arizona State Boxing Commission
00-19 Department of Economic Security—
Division of Developmental
Disabilities
00-20 Department of Corrections—
Security Operations
00-21 Universities—Funding Study
Future Performance Audit Reports
Department of Public Safety—Drug Abuse Resistance Education (D.A.R.E.) Program
Department of Economic Security—Division of Child Support Enforcement
Board of Osteopathic Examiners in Medicine and Surgery