Annual evaluation, Arizona's Family Literacy Program

              State of Arizona
Office
of the
Auditor General
PERFORMANCE AUDIT
Report to the Arizona Legislature
By Debra K. Davenport
Auditor General
March 2000
Report No. 00-3
ARIZONA'S
FAMILY LITERACY
PROGRAM
The Auditor General is appointed by the Joint Legislative Audit Committee, a bipartisan committee
composed of five senators and five representatives. His mission is to provide independent and impar-tial
information and specific recommendations to improve the operations of state and local government
entities. To this end, he provides financial audits and accounting services to the state and political
subdivisions and performance audits of state agencies and the programs they administer.
The Joint Legislative Audit Committee
Representative Roberta L. Voss, Chairman
Senator Tom Smith, Vice-Chairman
Representative Robert Burns Senator Keith Bee
Representative Ken Cheuvront Senator Herb Guenther
Representative Andy Nichols Senator Darden Hamilton
Representative Barry Wong Senator Pete Rios
Representative Jeff Groscost Senator Brenda Burns
(ex-officio) (ex-officio)
Audit Staff
Carol Cullen—Manager
and Contact Person (602) 553-0333
Lisa Townsend—Audit Senior
Copies of the Auditor General’s reports are free.
You may request them by contacting us at:
Office of the Auditor General
2910 N. 44th Street, Suite 410
Phoenix, AZ 85018
(602) 553-0333
Additionally, many of our reports can be found in electronic format at:
www.auditorgen.state.az.us
2910 NORTH 44th STREET • SUITE 410 • PHOENIX, ARIZONA 85018 • (602) 553-0333 • FAX (602) 553-0051
DEBRA K. DAVENPORT, CPA
AUDITOR GENERAL
STATE OF ARIZONA
OFFICE OF THE
AUDITOR GENERAL
March 2, 2000
Members of the Legislature
The Honorable Jane Dee Hull, Governor
Ms. Lisa Graham Keegan,
Superintendent of Public Instruction
Department of Education
Transmitted herewith is a report of the Auditor General, an evaluation of Arizona’s
Family Literacy Program. This is the fourth evaluation and was conducted pursuant to
the provisions of A.R.S. §41-1279.08. I am also transmitting with this report a copy of
the Report Highlights for this evaluation to provide a quick summary for your
convenience.
As outlined in its response, the Arizona Department of Education (ADE) agrees with all
of the findings. ADE agrees to implement the recommendations to take action against
sites that are repeatedly out of compliance and to provide additional technical
assistance to sites to test administration and reporting. However, ADE will implement
a different method of monitoring site compliance and reviewing sites.
My staff and I will be pleased to discuss or clarify items in the report.
This report will be released to the public on March 3, 2000.
Sincerely,
Debbie Davenport
Auditor General
Enclosure
i
OFFICE OF THE AUDITOR GENERAL
SUMMARY
The Office of the Auditor General has completed the fourth
evaluation of Arizona’s Family Literacy Program. This evaluation
was conducted pursuant to the provisions of A.R.S. §41-1279.08.
This evaluation provides information about the program’s effec-tiveness
and recommendations for the program.
The program’s intent is to improve the basic academic and liter-acy
skills of economically and educationally disadvantaged par-ents
and their preschool children. It is based on the premise that
the educational skills of parents, and, in turn, their children, must
increase in order to break the intergenerational cycles of poverty
and illiteracy. Arizona’s Family Literacy Program uses a model
that integrates the four main components advanced by the Na-tional
Center for Family Literacy. These components are adult
literacy instruction, parent and child together, parent education
and discussion and support groups, and early childhood educa-tion
for children ages 3 to 4. Families receive these services in a
classroom setting, generally on a school-year basis.
The State Board of Education and the Arizona Department of
Education (ADE) are responsible for administering Family Liter-acy.
To provide services, ADE contracts with other organizations,
including school districts, community colleges, and community-based
organizations. ADE currently has contracts with 13 organi-zations
to provide services at 23 sites. These sites are located in
five counties: Maricopa, Pima, Coconino, Yuma, and Cochise.
ADE Should Increase Program Oversight to
Improve Site Compliance with Program Criteria
(See pages 11 through 18)
ADE’s Adult Education Division should continue recent efforts to
increase program oversight to help ensure that participants are
receiving quality services. Although ADE administers the Family
Literacy Program, ADE has not used any of the state appropria-tion
for administration. As a result of this lack of funding, some
Summary
ii
OFFICE OF THE AUDITOR GENERAL
program sites are out of compliance with basic program stan-dards,
such as having a minimum enrollment of 10 families, and
integrating all 4 program components. Additionally, program
sites did not conduct all of the required tests of participants. As a
result, the amount and quality of available program information
for this evaluation is so limited that some program outcomes
cannot be assessed, and others are only partially assessed.
In July 1999, ADE committed additional resources to overseeing
the state-funded Family Literacy program and the Even Start
program, a similar federally funded family literacy program.
Nevertheless, ADE needs to take additional steps to strengthen
oversight and monitoring to help ensure that sites are in compli-ance
with all program, statutory, and contractual requirements.
Program’s Measurable Adult Education
Outcomes Are Generally Positive
(See pages 19 through 24)
Although lack of program information precludes assessing some
of the program’s adult education outcomes, analysis of those out-comes
that can be assessed shows the program is having at least
moderate positive results. In several respects, these results are
comparable to the results of a similar family literacy program, the
federally funded Even Start Program. Standard test scores in-creased
about the same for adults enrolled in English for Speakers
of Other Languages in the Family Literacy and Even Start pro-grams.
A similar analysis of test results could not be performed
on adult basic education and General Educational Development
(GED) instruction, because few participants took the required
pre- and posttests. However, available data on progress toward
completing the GED suggests that Family Literacy participants
make slower progress than Even Start participants do. Differ-ences
in participant demographic factors and program duration
could account for some of the differences in this outcome. Even
Start participants had more available time and resources than
Family Literacy participants had. Also, Even Start is a year-round
program, typically lasting three years, while Family Literacy is
limited to the regular school year and typically lasts only one
year.
Summary
iii
OFFICE OF THE AUDITOR GENERAL
Employment outcomes are about the same for the two programs;
employment rates increased from 14 percent at the beginning of
the 1998-99 school year to 34 percent at the end of the school year.
Adult Participants Show Improvements
in Parenting Attitudes and Behaviors
(See pages 25 through 31)
Adults enrolled in the parenting skills component of the Family
Literacy Program made improvements in their behaviors. Their
improvements in parenting behaviors were greater than their
improvements in parenting attitudes. Most parents entered the
program with favorable parenting attitudes and did not make
large improvements to their attitudes. However, parents who
entered the program with unfavorable parenting attitudes
showed large improvements in their parenting attitudes at the
end of the program. A comparison of gains made by participants
in the Even Start Program showed that both programs had es-sentially
the same impacts on parenting behaviors. However, the
Even Start Program had a greater impact on parenting attitudes.
Again, differences in participant demographics and program
duration could account for some of the differences in this out-come.
Preschool Participants Make Progress,
but Less Than Early Childhood
Block Grant Participants
(See pages 33 through 38)
In this evaluation, attempts were made to compare the school
readiness results of participants in the Family Literacy preschool
component with the results for participants in two similar pre-school
programs: the preschool portion of the federally funded
literacy program, Even Start; and the state-funded Early Child-hood
Block Grant preschool program. The latter is a stand-alone
preschool program rather than a family literacy program. At-tempts
to utilize scores from the Even Start comparison group
were thwarted because of inadequate data. Therefore, in this
evaluation, preschool readiness results from the Family Literacy
Program were compared only to the results from the Early
Childhood Block Grant preschool program.
Summary
iv
OFFICE OF THE AUDITOR GENERAL
Three areas in school readiness were measured: motor, problem-solving,
and language skills. The results show that the Family
Literacy Program’s preschool component continues to have an
impact on increasing children’s readiness to succeed in kinder-garten.
However, the state-funded Early Childhood Block Grant
preschool program had a greater impact on motor and problem-solving
skills. Differences in these two areas may be explained
partially by the age differences in the two groups: children in the
Early Childhood Block Grant were slightly older. At the beginning
of the 1998-99 school year, 28 percent of Family Literacy children
were under 4 years of age, as compared to only 3 percent of Early
Childhood Block Grant children. Although Family Literacy chil-dren
were younger than Early Childhood Block Grant children,
they were able to make equal gains in language skills.
v
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS
Page
Introduction and Background ......................... 1
Finding I: ADE Should Increase
Program Oversight to Improve
Site Compliance with Program
Criteria .......................................................... 11
ADE Was Unable to Provide
Sufficient Oversight to the
Family Literacy Program................................................ 11
Inadequate Data Submission and Collection
Limits Compliance with Program Requirements
and Ability to Measure Program Outcomes ................. 13
Partial Assessment Possible for
1998-99 Shows Compliance Problems
and Limits Outcome Assessment .................................. 13
ADE Has Taken Steps to
Improve Oversight ......................................................... 14
Data Collection System to
Enhance Oversight ......................................................... 16
Additional Steps Are Needed
to Improve Oversight..................................................... 17
Recommendations .......................................................... 18
Finding II: Program’s Measurable
Adult Education Outcomes
Are Generally Positive................................. 19
Adult Education Provided
in Two Basic Tracks ........................................................ 19
Table of Contents
vi
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS (Cont’d)
Page
Finding II (Cont’d)
Family Literacy Participants’ Rate
of English Language Skill Improvement
Matched Even Start Participants’ Rate .......................... 19
Complete Assessment Not Possible
for ABE/GED Participants............................................. 22
Participants’ Entry into Workforce
Is Comparable for Both Programs ................................. 24
Finding III: Adult Participants Show
Improvements in Parenting Attitudes
and Behaviors.............................................. 25
Program Seeks to Improve Both
Attitudes and Behaviors ................................................. 25
Parenting Attitudes Improve, but Not
as Much as Those of Even Start Participants'................ 27
Increases in Parenting Behaviors
Were Similar Across Programs ...................................... 28
Future Evaluations Will Continue to
Examine Trends of Family Literacy and
Even Start on Adult Outcomes...................................... 29
Finding IV: Preschool Participants Make
Progress, but Less Than
Early Childhood Block Grant
Participants .................................................. 33
Program’s Preschool Education
Component Aims to Enhance
Kindergarten Readiness ................................................. 33
Table of Contents
vii
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS (Cont’d)
Page
Finding IV: (concl'd)
Increases in Language Skills
Were Similar Across Programs ...................................... 34
Family Literacy Participants’
Problem-Solving and Motor Skills
Improve, but Not as Much as
Block Grant Participants’................................................ 35
Future Evaluations Will Continue to
Examine the Relative Effectiveness of the
Family Literacy Program in Increasing
Kindergarten Readiness Skills ....................................... 37
Statutory Evaluation
Components ..................................................... 39
Agency Response
Tables
Table 1 Family Literacy Program
Contractors, County, Type of Contractor,
and Number of Sites
1998-99 School Year........................................... 5
Table 2 Family Literacy Program
Contractors and Contract Amounts
Years Ended or Ending June 30, 1998
through June 30, 2000 (Unaudited).................. 7
Table 3 Family Literacy Program
Instances of Noncompliance by Site
and Requirement
1998-99 School Year........................................... 15
Table of Contents
viii
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS (Cont’d)
Page
Tables (Cont’d)
Table 4 Family Literacy Program
Changes in Parenting Behaviors
1998-99 School Year........................................... 29
Table 5 Family Literacy Program
Comparison to Even Start Program
Improvements in Adult Outcomes
1996-97 and 1998-99 School Year..................... 31
Table 6 Family Literacy Program
Early Success Evaluation
Comparison of Family Literacy and
Early Childhood Block Grant
Program Participants’ Improvements
in Average Test Scores
1998-99 School Year........................................... 37
Table 7 Family Literacy Program
Improvements in ChildOutcomes
Comparison to Even Start Program
1996-97 and Early Childhood Block
Grant Programs 1998-99 ................................... 38
Table 8 Family Literacy Program
Number and Percentage of Families
Completing Programs by Site
1996-97 and 1998-99 School Years
(Unaudited)....................................................... 43
Table of Contents
ix
OFFICE OF THE AUDITOR GENERAL
TABLE OF CONTENTS (Concl’d)
Page
Tables (Concl’d)
Table 9 Family Literacy Program
Number of Families Who Left the
Program and Reasons for Leaving
by Percentage
1998-99 School Year........................................... 44
Table 10 Family Literacy Program
Sites Ranked in Comparison with Each
Other by Improvement on Test
Measures 1998-99 School Year.......................... 50
Figure
Figure 1 Family Literacy Program
Language Assessment System Test Scores
1998-99 School Year........................................... 21
x
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
1
OFFICE OF THE AUDITOR GENERAL
Program’s Intent:
To improve the basic
academic and literacy
skills of undereducated
parents and their pre-school
children.
INTRODUCTION AND BACKGROUND
This is the fourth report issued by the Office of the Auditor
General evaluating Arizona’s Family Literacy Program. This
evaluation was conducted pursuant to the provisions of A.R.S.
§41-1279.08. This evaluation report provides information about
the program’s effectiveness and recommendations for the pro-gram.
Family Literacy Program
Aims to Improve Literacy
Skills of Parents and Children
The Legislature established the Family Literacy Pilot Program
in 1994 with legislation known as the Children and Families
Stability Act. The pilot program received an outcome evalua-tion
by the Office of the Auditor General in December 1997, and
was extended and renamed the Family Literacy Program by
Laws 1998, Ch. 295, §5, effective June 1, 1998.
The program is based on the
premises that:
4 Parents’ educational skills
must increase in order to
increase literacy and reduce
poverty among the current
generation of families; and
4 Children’s educational skills must increase in order to in-crease
literacy and reduce poverty among the next genera-tion
of families.
According to the National Center for Children in Poverty, chil-dren
whose parents lack a high school diploma are more than
twice as likely to live in poverty than children whose parents
Introduction and Background
2
OFFICE OF THE AUDITOR GENERAL
are high school graduates. By improving their literacy skills,
parents can improve employment options, consumer and fi-nancial
skills, and knowledge of health and personal safety
procedures.
The program is based on the recognition of the interdepend-ence
of children’s and parents’ education. Parents are their chil-dren’s
first and best teachers; thus, parents must cultivate and
value their own literacy skills in order to support their chil-dren’s
educational success. It is critical for parents to have ade-quate
literacy skills to promote their children’s healthy devel-opment
and acquisition of literacy.
Program Is Aimed at
Disadvantaged Parents
with Preschool Children
Family Literacy is directed at economically and educationally
disadvantaged parents with preschool-aged children. By stat-ute,
eligible parents must:
4 Have a three- or four-year-old child;
4 Lack sufficient mastery of basic educational or basic English
language skills to function effectively in society, or lack a
high school diploma or its equivalent; and
4 Be U.S. or legal residents, or otherwise lawfully present in
this country.
Families are recruited through a variety of methods, including
flyers, referrals from Head Start, social service agencies, previ-ous
participants, advertisements in local newspapers, and radio
announcements.
During the 1998-99 school year, program sites served a total of
402 families during all or part of the year. Sixty-two percent of
the families had incomes of $15,000 or less, and 29 percent of
the adults had less than a ninth-grade education. Adult partici-pants
had a median age of 28. Additional information about
Introduction and Background
3
OFFICE OF THE AUDITOR GENERAL
program participants is included in the section on statutory
evaluation components (see pages 39 through 50).
Program Model Includes Components
for Both Adults and Children
Family Literacy’s approach uses adult, child, and parent edu-cation
in an effort to be more effective than programs that focus
exclusively on adults or exclusively on children. Arizona’s
Family Literacy Program uses a model that integrates compo-nents
advanced by the National Center for Family Literacy.
Families receive services in a classroom setting, generally on a
school-year basis. Comprehensive literacy services are provided
by integrating the following four main components.
n Based on their individual needs, adults receive one or both
of the following adult literacy components:
4 Adult basic education and General Educational Devel-opment
(GED) preparation; and/or
4 Instruction in English for speakers of other languages.
n Early childhood instruction uses a developmentally appro-priate
curriculum to improve children’s motor, language,
and problem-solving skills. Teachers focus on:
4 Developing the cognitive, physical, social, and emo-tional
skills to improve children’s school readiness and
increase their chances of future academic success.
n Parenting education is provided through a component
called Parent and Child Together (PACT), which provides
opportunities for parents and children to play together or
work on structured activities. Parents learn to:
4 Teach and communicate with their children, using posi-tive
parenting skills that are transferable to the home.
Introduction and Background
4
OFFICE OF THE AUDITOR GENERAL
n Parent education discussion and support groups provide
opportunities for parents to learn from the experiences of
their peers, receive encouragement from the group, and
practice collective problem-solving. Together with PACT,
these discussion and support groups attempt to:
4 Provide parents with opportunities that will help them
learn how to best meet their children’s developmental
needs.
Program participants share some of the ways that the Family
Literacy program has impacted their lives:
Program Contractors Operate
23 Sites in 5 Counties
The State Board of Education and the Arizona Department of
Education (ADE) are responsible for administering Family Lit-eracy.
To provide program services, the ADE contracts with
other organizations, including school districts, a community
college, and community-based organizations.
“Thanks to Family Literacy and to my family for
their patience. I learned to listen more when my
daughter talks to me, before I heard her but I didn’t
listen to her. I feel my daughter is closer to me and
she trusts me more than last year. In other aspects
Family Literacy has given me the opportunity to
learn computers and improve my English. I also
know that Family Literacy gives me the confidence
I need to achieve my goals by myself.”
“I’m very happy because since my mom is in the
Family Literacy she helps me to do my homework.
She reads more books to us every night.”
“Learning the importance of reading to our chil-dren
everyday, helping the children with their
homework, valuing our families, gave me too many
opportunities to learn, that I can’t express on pa-per,
because I lived them everyday and you need to
live them in order to understand how I feel.”
(Comments from program participants)
Introduction and Background
5
OFFICE OF THE AUDITOR GENERAL
Table 1
Family Literacy Program
Contractors, County, Type of Contractor, and Number of Sites
1998-99 School Year
Contractor County
Type of
Contractor
Number
of Sites
Cochise Community College Cochise Community College 2
Flagstaff USD No. 1 Coconino School District 2
Literacy Volunteers of Maricopa
County Maricopa Community-based organization 2
Phoenix Indian Center Maricopa Community-based organization 1
Southwest Human Development Maricopa Community-based organization 1
Glendale ESD No. 40 Maricopa School District 1
Isaac ESD No. 5 Maricopa School District 2
Littleton ESD No. 65 Maricopa School District 1
Mesa USD No. 4 Maricopa School District 3
Tempe ESD No. 3 Maricopa School District 1
Pima County Adult Education Pima Adult education program 4
Crane ESD No. 13 Yuma School District 2
Somerton ESD No. 11 Yuma School District 1
Total 23
Source: Auditor General staff analysis of data provided by Family Literacy Program staff.
Programs are currently in place in five counties—Maricopa,
Pima, Coconino, Yuma, and Cochise. Contractors identify, re-cruit,
and screen participants and are mandated by legislation
to provide the following basic services:
n Instructional programs that promote academic and literacy
skills and equip parents to provide needed support for their
children’s educational growth and success.
n A plan to address program participants’ transportation,
food, and childcare needs during the program.
n An organizational partnership involving (at a minimum) a
common (public) school, a private preschool provider, and
an adult education program funded by ADE.
Family Literacy had 13 contractors during the 1998-99 school
year, as shown in Table 1. In all, these 13 contractors adminis-tered
a total of 23 sites, 12 of them in Maricopa County. At
each site, contractors are required to enroll between 10 and 20
participants. Two of the contractors, Mesa Unified School Dis-trict
and Pima County Adult Education, operate “model pro-
Introduction and Background
6
OFFICE OF THE AUDITOR GENERAL
grams” that provide technical assistance to all sites and training
to family literacy staff.
The Legislature appropriated $1 million for each of the four
school years from 1996 to 2000. ADE does not retain any funds
for administration. All monies appropriated for the Family Lit-eracy
Program are awarded directly to sites. Any leftover funds
are nonreverting and can be carried over to the next year.
Resources required to operate a family literacy program vary
depending on the number of days per week the program oper-ates,
the program size, and the cost of conducting the program
in a particular community. The National Center for Family Lit-eracy
(NCFL) estimates that one full-time program requires
$50,000 to $90,000 per year in a rural area and $75,000 to
$125,000 in an urban area. Many of the state family literacy sites
have funding collaborators who provide financial support in
addition to the state grant.
See Table 2 (page 7) for information on contract amounts dur-ing
fiscal years 1998 through 2000.
Follow-Up of Previous
Evaluation Reports
  Compliance with Program Model—The previous three
evaluation reports (Auditor General Report No. 95-20, Re-port
No. 96-20, and Report No. 97-22) all addressed pro-gram
implementation. Although the 1995 evaluation found
that the pilot program used a model that followed recom-mended
standards, the 1996 evaluation found that many
sites failed to comply with these standards. The 1997
evaluation found that most sites subsequently improved
their compliance with the program model, and three sites
that continued to have implementation problems were to be
closely monitored by ADE to ensure that they met program
guidelines. In this fourth evaluation report, compliance
Introduction and Background
7
OFFICE OF THE AUDITOR GENERAL
Table 2
Family Literacy Program
Contractors and Contract Amounts
Years Ended or Ending June 30, 1998 through 2000
(Unaudited)
Contractor 1998 1999 2000
Apache County
Red Mesa USD No. 27 $ 66,387
Cochise County
Cochise Community College 1 33,419 $ 89,200 $ 89,200
Coconino County
Flagstaff USD No. 1 (2 sites) 56,540 59,700 59,700
Maricopa County
Glendale ESD No. 40 40,000 40,000
Isaac ESD No. 5 ( 2 classes) 66,438 71,400 71,400
Literacy Volunteers of Maricopa County1 30,248 67,900 67,900
Littleton ESD No. 65 40,000 40,000
Mesa USD No. 4 (3 sites and a model program) 153,148 168,100 169,600
Phoenix Indian Center 32,642 36,600 36,600
Southwest Human Development 30,100 30,100
Tempe ESD No. 3 33,220 38,200 38,200
Navajo County
Pinon USD No. 4 91,523
Pima County
Pima County Adult Education2 167,003 222,000 226,500
Tucson USD No. 1 (2 sites) 165,968
Yuma County
Crane ESD No. 13 ( 2 classes) 56,824 70,300 70,300
Somerton ESD No. 11 46,508 46,500 46,500
Total $999,868 $ 980,000 $986,000
1 Contractor had one site in 1998 and two sites in 1999 and 2000.
2 Contractor had three sites and a model program in 1998 and four sites and a model program in 1999 and
2000.
Source : Auditor General staff analysis of data provided by the Arizona Department of Education.
with the program model was again addressed, and consider-able
variation between sites was found.1 (See Finding I, pages
11 through 18.)
1 This Office completed the evaluation requirements under the pilot legisla-tion
in 1997. Subsequent legislation, effective June 1, 1998, provided new
evaluation requirements. This is the first report issued under the new
evaluation requirements.
Introduction and Background
8
OFFICE OF THE AUDITOR GENERAL
n Program Funding—The 1995 and 1996 evaluations re-ported
on the impact of a budgeting error that resulted in a
need to reduce program expenditures. Follow-up in this
current report showed that the need to reduce expenditures
was no longer present.
n Inadequate Monitoring—The 1995 and 1996 evaluations
identified inadequate program monitoring and oversight by
ADE as a reason for inadequate compliance with the pro-gram
model. The 1997 evaluation found that ADE had sub-sequently
made great improvements in program monitor-ing
and providing technical assistance. Follow-up in this
current report showed that program monitoring had again
surfaced as a problem (see Finding I, pages 11 through 18).
Scope and Methodology
Methods used in this evaluation include analyses of tests at the
beginning and end of the program, structured observations,
and site reviews. Analyses were performed on reported out-comes
for families in the program as noted on exit forms and by
comparing scores at the end of the school year to those at the
beginning of the school year on:
n Results of standardized tests designed to measure
a) adult basic education skills,
b) English language skills, and
c) parenting skills;
n Adult’s parenting behavior scores; and
n Children’s development scores.
All data analyzed are from the 1998-99 school year only; insuffi-cient
data was available from the 1997-1998 school year (see
Finding I, pages 11 through 18). During the 1998-99 school year,
structured site observations were conducted at 20 sites. As part of
these visits, program trainings were observed and site documents
were reviewed.
Introduction and Background
9
OFFICE OF THE AUDITOR GENERAL
For this evaluation report, Family Literacy participants were
tested to determine if they improved literacy skills. Participants
from the 1998-99 school year took standardized tests upon enter-ing
the Family Literacy Program and at the end of the school
year. They were given the Language Assessment System to assess
their English language reading and writing skills and the Compre-hensive
Assessment System of Academic Skills to assess their English
language listening comprehension.
To assess how these results compare with those of another type of
family literacy program, the results were compared to those of
participants in the 1998-99 Even Start Program. Even Start is a
federally funded family literacy program that is a national leader
in family literacy. Both programs adhere to models that integrate
components advanced by the National Center for family Literacy
(NCFL), though they do have some differences in such charac-teristics
as age of participants, length of the instructional year, and
other factors.1 Even Start participants were administered pre- and
posttests using the same instruments.
Attempts were also made to form a comparison group of adult
education participants, which would allow for an assessment of
how effective the state family literacy program is in contrast to
stand-alone programs. These efforts were unsuccessful.
The preschool component was compared to the 1998-99 Early
Childhood Block Grant program. The Early Childhood Block
Grant program is a state-funded preschool program for at-risk
children. Early Childhood Block Grant participants were admin-istered
pre- and posttests using the same instrument as the Fam-ily
Literacy participants, the Early Success Evaluation (ESE). Family
Literacy children’s ESE scores were compared to a group of Early
Childhood Block Grant children’s scores. The Block Grant group
was matched to the Family Literacy group by race and primary
language spoken.
1 For example, Even Start allows children to be age 0-7 (and in some
cases up to 10), while Family Literacy restricts eligibility to 3- and 4-
year-olds. Even Start programs operate on a year-round basis, while
most Family Literacy programs operate only during the school year.
Introduction and Background
10
OFFICE OF THE AUDITOR GENERAL
Attempts were also made to form a comparison group of Even
Start preschool participants, which would allow for an assess-ment
of how effective the state family literacy preschool compo-nent
is in contrast to other family literacy programs. These efforts
were unsuccessful because of inadequate Even Start ESE data.
This report again addresses issues related to compliance with
program criteria (see Finding I, pages 11 through 18). It also fo-cuses
on the Family Literacy Program’s effectiveness in improv-ing
the following:
n English language skills and education and employment abili-ties
of adult participants (see Finding II, pages 19 through 24);
n Parenting skills of adult participants (see Finding III, pages 25
through 31); and
n School readiness of the three- and four-year-old preschoolers
in the program (see Finding IV, pages 33 through 38).
Acknowledgements
The Auditor General and staff express appreciation to the Su-perintendent
of Public Instruction, staff of the Arizona Depart-ment
of Education’s Adult Education Division, and the Family
Literacy and Even Start Programs’ staff and families for their co-operation
and assistance.
11
OFFICE OF THE AUDITOR GENERAL
FINDING I ADE SHOULD INCREASE
PROGRAM OVERSIGHT TO
IMPROVE SITE COMPLIANCE WITH
PROGRAM CRITERIA
ADE’s Adult Education Division should continue recent efforts
to increase program oversight to help ensure that participants
are receiving quality services. Following the last program
evaluation by the Auditor General’s Office in 1997 (Report No.
97-22), the extent of monitoring declined. As a result, some pro-gram
outcomes cannot be assessed at all, and others can be as-sessed
only partially. In addition, a number of program sites are
out of compliance with basic program standards, such as
minimum enrollments and component integration. In July 1999,
ADE committed additional resources to overseeing both the
state-funded Family Literacy Program and the federally funded
Even Start Program. Nevertheless, ADE needs to take addi-tional
steps to strengthen oversight by ensuring that all sites
receive structured reviews and by using its model programs to
help other sites administer tests and report scores correctly.
ADE Was Unable to Provide
Sufficient Oversight to the
Family Literacy Program
ADE’s administrative oversight of Family Literacy sites deterio-rated
during school years 1997-98 and 1998-99. Previous
evaluation reports had pointed out problems with the extent of
ADE’s monitoring, and by the 1996-97 school year, improve-ments
had been made. Early in the program, an over-awarding
of money to contractors subsequently caused budget cuts that
eliminated administrative funding. Despite the lack of admin-istrative
funding, ADE improved oversight during the 1996-97
school year, which brought increased compliance with program
requirements. However, these oversight efforts did not carry
over into the following years.
Finding I
12
OFFICE OF THE AUDITOR GENERAL
ADE lacks administrative funding for the Family Literacy Pro-gram—
Although the Family Literacy Program is administered
by ADE, ADE has not used any of the state appropriation for
administration. Therefore, in the past the Division of Adult
Education has used Federal Leadership funds to assist in pro-gram
administration. Recent changes in federal legislation have
reduced the availability of this fund. ADE estimates that 10 per-cent
of the state appropriation is needed for thorough admini-stration
of the Family Literacy program.
Lack of administrative funding lessened ADE's ability to pro-vide
adequate oversight to the Family Literacy Program—
ADE's data collection for the Family Literacy Program was
weakened by a lack of administrative oversight over the
program during the 1997-98 and 1998-99 school years. Matters
of particular concern, because of their impact on the program,
are as follows:
n During the 1997-98 school year, when the Auditor General
was not required to evaluate the program, ADE was solely
responsible for collecting program data. These efforts were
not adequate. Of the 21 sites in operation during that year,
ADE collected intake information from 18 sites and exit in-formation
from only 9. Even among these sites, information
was not complete. For example, only 9 sites provided com-plete
parent pre- and posttest data, while 8 provided only
pre-test data. Only 3 sites provided complete child pre- and
posttest data, while 5 others provided only pretest data.
n During the 1998-99 school year, ADE completed structured
visits with 5 of 23 sites operated by 13 contractors (also
called "programs" by ADE). These visits involve such mat-ters
as budget and expenditure reviews, checks for compli-ance
with legislated program requirements, and reviews of
program records. Although ADE did perform less-structured
site visits at 17 sites, the information collected
and the degree of compliance review was, again, only par-tial.
1
1 ADE did not visit one site at all (Leupp, under the Flagstaff contractor).
This site has had problems meeting enrollment and eligibility require-ments
since the 1995-96 school year.
Finding I
13
OFFICE OF THE AUDITOR GENERAL
Inadequate Data Submission
and Collection Limits Compliance
with Program Requirements and Ability
to Measure Program Outcomes
Sites did not report enough information to allow any evaluation
of outcomes for 1997-98. Evaluators were able to gather some
information for the 1998-99 school year showing that compli-ance
problems remained. Some of these compliance problems
limited evaluators’ ability to assess outcomes for 1998-99.
n No Assessment Possible for 1997-98—Although the
Auditor General is required to evaluate the Family Literacy
Program only for the 1998-99 school year, previous evalua-tion
reports compared program information from the cur-rent
year to previous years. Because sites did not submit
data as stipulated in their contract and ADE did not collect
or was not able to collect complete program data for the 1997-
98 school year, this evaluation could not make such compari-sons
between the 1997-98 and 1998-99 school years. The
Auditor General does compare outcomes in this evaluation to
the outcomes found in the 1997 evaluation.
Partial Assessment Possible for
1998-99 Shows Compliance Problems
and Limits Outcome Assessment
In addition to problems with data submission and collection
during the 1997-98 school year, the 1998-99 school year data was
also incomplete in many instances. Because ADE’s monitoring
had been so limited, the available information was not complete
enough to allow some aspects of the program to be assessed. The
available information indicates that statutory and program com-pliance
problems exist at some sites.
Noncompliance with testing procedures limits outcome as-sessment—
Due to a lack of compliance with test administration
and reporting requirements, this evaluation cannot report on
certain program outcomes. Contracts require the sites to report
information on student progress to ADE. However, sites did
not administer the Tests of Adult Basic Education (TABE) to most
Finding I
14
OFFICE OF THE AUDITOR GENERAL
adult basic education and GED students. Because so few stu-dents
took this test, evaluators were unable to determine the
outcomes for participants. Project directors or their staff did not
verify that the intake forms contained complete information
from both adult and child participants. Further, one site failed
to administer the Early Success Evaluation (ESE) test to child
participants. Finally, ADE did not verify that forms submitted
by contractors were complete.
Compliance problems exist in key areas—In addition to testing
compliance problems, sites experienced compliance problems
with enrollment and component integration, as Table 3 (see
page 15) shows. Areas of particular concerns are as follows:
n Enrollment—Four sites were out of compliance with mini-mum
enrollment requirements throughout the school year.
Although sites are statutorily required to enroll between 10
and 20 families, 4 sites did not maintain a minimum enroll-ment
of 10 families for the majority of the school year.
n Component Integration—Sites are required to integrate the
four Family Literacy Program components: adult literacy
instruction, early childhood instruction, parent and child
interaction time, and parent discussion and support group.
Four of the 23 sites, including one that had experienced
problems in the past, were not in compliance with this re-quirement.
ADE Has Taken Steps
to Improve Oversight
ADE has made some improvements in its oversight of the
Family Literacy Program. Besides continuing to offer profes-sional
development activities, ADE also created a full-time
Family Literacy Program Coordinator position beginning in
July 1999.
Professional development opportunities continue—As part of
its oversight role, ADE provides opportunities for program staff
Table 3
Family Literacy Program
Instances of Noncompliance by Site and Requirement
1998-99 School Year
Site Enrollment Eligibility
NCFL
Components
Components
Integration
Child
Care Transportation Total
Cochise College
Sierra Vista a 0
Douglas 0
Crane ESD No. 13
Crane AM Session 0
Crane PM Session 0
Flagstaff USD No. 1
Leupp X b X 2
Killip 0
Glendale ESD No. 40 a X 1
Isaac ESD No. 5
Isaac English for Speakers of Other Languages 0
Isaac GED 0
Literacy Volunteers of Maricopa County
Montecito a X X 2
Wesley a X X 2
Littleton ESD No. 65 a X 1
Mesa USD No. 4
Eisenhower 0
Longfellow 0
Lincoln 0
Phoenix Indian Center X 1
Pima County Adult Education
Liberty 0
Nash 0
South Tucson a 0
Prince a 0
Somerton ESD No. 11 X 1
Southwest Human Development a 0
Tempe ESD No. 3                       0
Total 4 1 1 4 0 0 10
a New sites in 1998-99.
b Leupp does not meet enrollment requirements when ineligible participants are excluded from the count.
Source: Auditor General staff analysis of data provided by Arizona Department of Education and Family Literacy program staff.
Finding I
OFFICE OF THE AUDITOR GENERAL 15
Finding I
16
OFFICE OF THE AUDITOR GENERAL
and administrators to increase their knowledge of topics related
to Family Literacy:
4 A statewide Family Literacy conference is held annually for
all Family Literacy staff. Over 200 teachers, administrators,
staff personnel, and volunteers registered for the January
1999 conference, with representation from all Arizona Fam-ily
Literacy programs.
4 Also, model programs annually conduct a Family Literacy
Implementation training to orient new Family Literacy Pro-gram
staff.
More staff resources devoted to oversight—ADE created a po-sition
that has a direct bearing on the amount of oversight and
monitoring in the Family Literacy Program. A staff member
was hired in March 1999 as an Education Program Specialist
with a focus on Family Literacy. This staff member’s ability to
provide oversight assistance to the Family Literacy program
was initially limited because she was also responsible for
monitoring the adult education program. In July 1999, ADE
changed the position title to Family Literacy Coordinator. This
new position is intended to provide oversight to both the state
Family Literacy program and the federal Even Start program.
Data Collection System
to Enhance Oversight
ADE is currently developing an Internet version of a student
information management system that will allow it to track stu-dent
progress. ADE will link the information about Family Lit-eracy
participants to the State’s Student Accountability Infor-mation
System (SAIS), which tracks K-12 student information.
In addition to monitoring student progress, an advantage of the
Internet data collection system is that it will provide uniformity
in Family Literacy sites’ reporting of student information.
Finding I
17
OFFICE OF THE AUDITOR GENERAL
Additional Steps Are Needed
to Improve Oversight
Although the addition of a position to oversee Family Literacy
represents a positive step, other actions would help ensure that
oversight quality is improved and made more consistent. Spe-cifically,
ADE needs to ensure that sites receive more compre-hensive
oversight visits. During these visits, ADE needs to pro-vide
technical assistance on administering tests to participants.
More structured site reviews needed—During the 1998-99
school year, only five sites received a structured site review. All
Family Literacy Program sites should receive such a review.
Site visits should involve:
4 On-site contact;
4 Structured classroom observation;
4 Budget and expenditure reviews;
4 Checks for compliance with legislated program require-ments;
4 Review of data collection on student enrollment and atten-dance,
student progression, and other statistical information
required to be reported to the State; and
4 Follow-up when specific deficiencies are identified.
Since sites were out of compliance with test administration and
reporting requirements, ADE should also include a review of
test documentation during site visits.
Test administration needs more focus—During site visit inter-views
by Auditor General staff, several site staff requested
more training in test administration and reporting require-ments.
ADE has two “model programs” (Pima County Adult
Education and Mesa Family Tree), that have provided training
workshops on testing at the beginning of the school year. While
these model programs should continue to provide such assis-tance,
ADE should increase the focus on test administration
Finding I
18
OFFICE OF THE AUDITOR GENERAL
during site visits. As testing and reporting requirements are key
to demonstrating the program’s effectiveness, ADE should
check to determine that tests are being administered correctly
and that scores are reported correctly during site visits to family
literacy programs.
Recommendations
1. The Arizona Department of Education (ADE) should ac-tively
and continuously monitor sites each year for compli-ance
with program requirements.
2. ADE should conduct comprehensive, structured site re-views
with each site at least once a year to ensure sites are in
compliance with all statutory, program, and contractual re-quirements.
3. ADE should take action against sites that are out of compli-ance
with statutory, program, or contractual requirements.
4. ADE should provide additional technical assistance to sites
related to test administration and reporting.
19
OFFICE OF THE AUDITOR GENERAL
FINDING II PROGRAM’S MEASURABLE
ADULT EDUCATION OUTCOMES
ARE GENERALLY POSITIVE
Although lack of data precludes assessing some of the program’s
adult education outcomes, analysis of those outcomes that can be
assessed shows that the program demonstrates at least moderate
results. In several respects, these results are comparable to the
results of Even Start, a federally funded family literacy program.
Results are most readily measurable in the teaching of English
for speakers of other languages (ESOL), where scores on stan-dardized
tests increased about the same in both programs. A
similar analysis of test results could not be performed on adult
basic education (ABE) and General Educational Development
(GED) instruction, because few participants took required pre-and
posttests. However, available data on progress toward com-pleting
the GED suggests that Family Literacy participants make
slower progress than Even Start participants. Employment out-comes
are about the same for the two programs.
Adult Education Provided
in Two Basic Tracks
When they enter the Family Literacy Program, adult participants
enroll in one or both of two educational tracks (ESOL or
ABE/GED), depending on their goals. Of the adult education
participants who completed exit forms during the 1998-99 school
year, 218 indicated that they participated in the ESOL compo-nent,
while 170 indicated that they participated in the ABE/GED
component.
Family Literacy Participants’ Rate of
English Language Skill Improvement
Matched Even Start Participants’ Rate
The adults who participated in the ESOL component of Family
Literacy in the 1998-99 school year made statistically significant
Finding II
20
OFFICE OF THE AUDITOR GENERAL
gains in reading, writing, and listening comprehension skills,
though the gains in reading and writing skills were higher.
Their results were comparable to improvements made by
adults in the Even Start program.
Reading and writing skills improve—Participants increased
their English reading and writing skills during the 1998-99
school year, as measured by reading, writing, and the com-bined
reading/writing score on the Language Assessment
System (LAS) test. In all, 149 adults had pre- and posttest scores
for the LAS. Scores can range from a low of 0 to a high of 100.
As Figure 1 (see page 21) shows, on this 100-point scale the
1998-99 participants’ average reading gain was approximately
10 points and the average writing gain was approximately 16
points. Scores are grouped into 5 categories of competence:
4 Low beginner (0 to 20);
4 High beginner (21 to 40);
4 Low intermediate (41 to 60);
4 High intermediate (61 to 80); and
4 Competent (81 to 100).
On average, participants moved their reading scores from low
intermediate to high intermediate and their writing scores from
high beginner to low intermediate. The gains were statistically
significant.
Smaller gains in listening comprehension—While participants
improved their English listening comprehension skills, the
gains were smaller than those for reading and writing skills.
Listening comprehension skills are measured using the Com-prehensive
Assessment System of Academic Skills (CASAS).
The 136 adults who took the CASAS increased their scores an
average of 5 points. Although the CASAS scores increased, they
remained in the “intermediate” category of competence. At the
intermediate level, students can “satisfy basic survival needs
and a very few routine social demands.” Once again, the gains
were statistically significant.
Finding II
21
OFFICE OF THE AUDITOR GENERAL
Family Literacy results similar to Even Start participants and
to 1997 evaluation—Even Start participants across Arizona
were assessed using the same tests as Family Literacy partici-pants.
No significant differences in gains were found between
the Family Literacy and Even Start participants, either on LAS
or CASAS results. This suggests that the state program is hav-ing
the same level of success in this area as the federal program.
The results are also comparable to the Auditor General’s 1997
evaluation.
Figure 1
Family Literacy Program
Language Assessment System Test Scores
1998-99 School Year
0
10
20
30
40
50
60
70
a The improvements are statistically significant at the .01 level. That is, the probability that the
average gain occurred by chance is less than 1 in 100.
Source: Auditor General staff analysis of data provided by Family Literacy Program staff.
33.83 49.58
Improvement = 9.48a
Improvement = 15.75a
Reading Writing
Pretest Posttest
Score
54.43 63.91
Finding II
22
OFFICE OF THE AUDITOR GENERAL
Complete Assessment Not Possible
for ABE/GED Participants
Due to the lack of test data available from Family Literacy sites,
complete assessment was not possible for those participants
enrolled in ABE/GED. Information on participants’ progress
toward the GED indicates that they proceed more slowly than
participants in the Even Start program. Certain differences be-tween
the two programs may help explain this result.
Basic education outcomes unknown for ABE/GED students—
Progress among ABE/GED students is measured using the
Tests of Adult Basic Education (TABE), which measures
achievement in mathematics, language, reading, and spelling.
Sites should have administered the TABE pre- and posttest to
all adult basic education and GED students. However, it is not
possible to report on adult basic education outcomes for the
adults who participated in the ABE/GED component during
the 1998-99 school year because too few students took the
TABE. Of the 170 participants who received ABE and GED
services, only 28 took both the pretest and the posttest.1
Family Literacy participants’ progress toward attaining GED
is slower than Even Start participants’—While sites did not
collect sufficient data to allow an evaluation of progress on
standard tests, they did collect information on the number of
adults who received their GED at the end of the school year. Of
the 115 adults who answered the exit form question regarding
their participation in the ABE/GED component during the
1998-99 school year, 4 received their GED or high school di-ploma
since they began participating in the Family Literacy
Program. Twelve others were in the process of taking the GED
tests at the end of the school year but had not yet taken or
passed all of them, and 90 more reported that they were work-ing
toward their GED at the end of the school year. Two adults
who participated in the ABE/GED component during the 1998-
99 school year have continued on to college.
1 The 1997 evaluation found that TABE scores increased for participants in
the ABE/GED component.
Finding II
23
OFFICE OF THE AUDITOR GENERAL
A comparison of results from Even Start participants showed
that Even Start participants were more likely than Family Liter-acy
participants to:
Have obtained their GED;
Be in the process of taking the test; or
Be attending a community college or university.
Thirty-one percent of Even Start participants achieved these out-comes,
compared to 16 percent of Family Literacy participants.
The difference between the two programs on GED and higher
education outcomes is statistically significant.
Certain differences in program participants and program char-acteristics
may help explain these results.
n Demographic differences—There are a number of demo-graphic
factors that could account for the differences between
the Family Literacy and Even Start programs on GED and
higher education outcomes. The Family Literacy participants
may have less time and resources to devote to studying for
and completing their GED. Specifically:
Twice as many Family Literacy participants (10 percent)
as Even Start participants (5 percent) are single parents
with children.
Family Literacy participants had slightly less average in-come
than Even Start participants did. Thirty-six percent
of Family Literacy participants made less than $10,000 a
year, in contrast to 23 percent of Even Start participants.
Also, more Family Literacy participants (13 percent) were
working full-time while they were enrolled in the pro-gram
than Even Start participants (7 percent).
Finding II
24
OFFICE OF THE AUDITOR GENERAL
n Program duration differences—Even Start participants
may be more successful at attaining their GED because the
Even Start Program engages participants for a longer dura-tion
than the Family Literacy Program does.
Participants’ Entry into Workforce
Is Comparable for
Both Programs
Although many participants are focusing on raising families
and therefore not seeking employment, during the 1998-99
school year, Family Literacy participants entered the workforce
at the same rate as Even Start participants. At the beginning of
the 1998-99 school year, 18 percent of participants were em-ployed
either full-time or part-time. By the end of the school
year, that figure had grown to 34 percent. There was no signifi-cant
difference in employment outcomes between Family Liter-acy
and Even Start participants, which is similar to the results
found in the 1997 evaluation.
Even Start Program
4 Provides services on a year-round
basis for a minimum
of three years.
4 Participants are eligible to
participate with a child from
birth up to age 10.
4 Participants participated in
the program for a median of
224 hours.
Family Literacy Program
4 Provides services during the
regular school year only
(September through May);
4 Participants must have a 3-
or 4-year-old-child to re-main
eligible for the pro-gram;
and
4 Participants participated in
the program for a median of
144 hours.
25
OFFICE OF THE AUDITOR GENERAL
FINDING III ADULT PARTICIPANTS SHOW
IMPROVEMENTS IN PARENTING
ATTITUDES AND BEHAVIORS
Adults enrolled in the parenting skills component of the Family
Literacy Program improved both their attitudes about parent-ing
and the parenting behaviors they reported. Most parents
entered the program with favorable parenting attitudes and
made small improvements; parents who entered the program
with unfavorable parenting attitudes improved the most. On
average, however, the federally funded Even Start Program
had a greater impact on parenting attitudes. Both programs had
essentially the same impact on parenting behaviors.
Program Seeks to Improve
Both Attitudes and Behaviors
The Arizona Family Literacy Program focuses on developing
parenting skills through two components:
Parent and Child Together (PACT), and
Parenting group discussion.
These components are intended to help parents help their chil-dren
learn through play and more structured activities and
provide parents with opportunities to learn how to best meet
their young children’s developmental needs. These components
are intended to develop both:
4 Positive attitudes about parenting; and
4 Positive parenting behaviors.
To assess the extent to which the Family Literacy Program im-pacted
adults’ parenting attitudes and behaviors, participants
Finding III
26
OFFICE OF THE AUDITOR GENERAL
were pre- and posttested on two instruments. Even Start par-ticipants
were given the same pre- and posttest instruments to
determine if there was any difference in the two programs’ ef-fectiveness.
n Attitudes—Attitudes were measured by using the Parent as
a Teacher Inventory (PAAT). PAAT identifies favorable
qualities and realms in which personal growth is needed in
regard to parenting children ages 3 to 9. PAAT items in-clude
a variety of statements on:
What parents want or expect of their child,
How they interact with their child, and
What actions are taken in response to specific child be-havior.
The responses are grouped into five areas with subscores
computed for each area and a total score. The five areas are:
Creativity—parental acceptance of the child’s creativity
and willingness to encourage its development;
Frustration—parental frustration with the child and fo-cus
of the frustration;
Control—parental feelings about the need to control the
child’s behavior;
Play—parental understanding of play and its influence
on child development; and
Teaching/Learning—parental views about child devel-opment
and their ability to provide a supportive home
environment.
n Behaviors—The parents’ behavior was measured by using
a Behavior Frequencies Assessment (BFA), which was de-veloped
specifically for the Family Literacy Program
evaluation. The BFA is a self-report instrument on which
parents indicate how frequently they engage in ten different
Finding III
27
OFFICE OF THE AUDITOR GENERAL
behaviors, such as reading to their children and helping
their children with homework.
Parenting Attitudes Improve,
but Not as Much as Those of
Even Start Participants’
Family Literacy participants improved their parenting atti-tudes,
but not as much as Even Start participants improved.
Most Family Literacy parents made modest gains in parenting
attitudes. However, the parents who began the Family Literacy
program with more unfavorable parenting attitudes made
larger gains. Average improvement in parenting attitudes was
greater for Even Start participants than for Family Literacy par-ticipants.
n Most parents show modest gains in parenting atti-tudes—
Family Literacy parents entered the program with
good attitudes overall. The 213 parents who were pre- to
posttested during the 1998-99 school year entered with an
average pre-test score of 138.6 on a scale of 0 to 200, well
above the 125-point cutoff for “favorable” parenting atti-tudes.
They made small but statistically significant im-provements
in their total score, increasing it to an average of
143.3. This outcome was similar to results reported in the
1997 evaluation.
n Parents entering with unfavorable attitudes show the
largest improvement—Parents who entered the program
with unfavorable parenting attitudes made the largest gains
on the PAAT. The 13 parents who entered the Program with
PAAT scores below 125 points gained an average of 19
points. Their average pretest score was 120.77, and their av-erage
posttest score was 139.23.
n Average improvement greater for Even Start partici-pants—
Comparison of PAAT results for Family Literacy
Program participants and Even Start participants shows
that Even Start had greater impact on parenting attitudes.
Even Start participants’ average scores were 136.08 on the
pretest and 144.45 on the posttest, an improvement close to
Finding III
28
OFFICE OF THE AUDITOR GENERAL
4 points higher than that of Family Literacy participants.
The difference between the two groups was statistically sig-nificant.
Even though both groups were initially scoring in
the “favorable” range, this suggests that the federal pro-gram
has a higher rate of success improving parenting atti-tudes
than the state program. This outcome differs from the
1997 evaluation, which found no statistically significant dif-ference
between Family Literacy and Even Start partici-pants’
results.
Increases in Parenting Behaviors
Were Similar Across Programs
Family Literacy parents made statistically significant improve-ment
in 9 out of 10 of the BFA measures from pre- to posttest,
making the biggest gains in the following areas:
4 Going to school activities,
4 Talking to their children’s teachers,
4 Volunteering to help with school activities,
4 Helping their children with homework, and
4 Talking to their children about school.
Of these, the highest scores were in helping children with
homework and talking to children about school (see Table 4,
page 29). It appears that the program is most useful in increas-ing
parents’ involvement with their children in activities di-rectly
related to school.
Comparison of changes for Family Literacy Program partici-pants
and Even Start participants on the BFA shows that both
programs have essentially the same impacts on parenting be-haviors.
Both groups showed improvement from pre- to post-test.
There are no statistically significant differences between
the two groups on any of the BFA subscore pre- to posttest re-sults.
Finding III
29
OFFICE OF THE AUDITOR GENERAL
Table 4
Family Literacy Program
Changes in Parenting Behaviors
1998-99 School Year
Number of Days
Behavior Beginning End Improvement
Number of days per week parents:
Talk about children’s day 5.40 5.84 .44a
Read or look at book with children 4.25 4.65 .45b
Let children see them writing 4.47 4.86 .39b
Take children to the library 0.85 1.23 .38a
Play with children 5.74 5.78 .04c
Number of days per month parents:
Go to a school activity 1.70 2.85 .15b
Volunteer for a school activity 2.17 4.56 2.39a
Help children with homework 10.22 13.69 3.47a
Talk to children’s teacher 3.24 5.46 2.22a
Talk to children about school 11.72 16.56 4.84a
a The improvements are statistically significant at the .01 level. That is, the probability that the average gain
occurred by chance is less than 1 in 100.
b The improvements are statistically significant at the .05 level. That is, the probability that the average gain
occurred by chance is less than 5 in 100.
c Not statistically significant.
Source: Auditor General staff analysis of data provided by Family Literacy Program staff.
Future Evaluations Will Continue to
Examine Trends of Family Literacy and
Even Start on Adult Outcomes
With the exception of two outcomes, the Family Literacy and
Even Start programs are beginning to show a consistent and
similar pattern of results as they pertain to adult-related out-comes.
However, because the levels of oversight and program
compliance differed during the two periods in which the
Finding III
30
OFFICE OF THE AUDITOR GENERAL
evaluations were conducted, the next evaluation conducted by
the Office of the Auditor General will examine trends and out-comes
as they relate to levels of oversight and program compli-ance.
Overall, this evaluation found that adult-related outcomes for
Family Literacy participants are moderately positive and simi-lar
to Even Start adults’ performance in three areas:1
n Development of English language skills;
n Entry into the workforce; and
n Development of parenting behaviors.
However, in contrast to the 1997 evaluation, there were two
areas in which the Even Start participants’ performance ex-ceeded
the Family Literacy participants’ performance (see Table
5, page 31). These areas were:
n Progress toward attaining a GED; and
n Development of parenting attitudes.
Future evaluations will examine whether greater oversight and
program compliance on the part of the Family Literacy pro-gram
will improve the adult-related outcomes.
1 Adult basic education skills were not assessed in this current
evaluation due to lack of data. The 1997 evaluation found that Fam-ily
Literacy participants’ performance in adult basic education was
positive and similar to the Even Start participants’ performance.
Finding III
31
OFFICE OF THE AUDITOR GENERAL
Table 5
Family Literacy Program
Comparison to Even Start Program Improvements in Adult Outcomes
1996-97 and 1998-99 School Years
Areas of Adult Outcomes 1996-97 1998-99
Adult Basic Education Comparable Unable to determine 1
General Educational Development Comparable Less than Even Start
English language skills Comparable Comparable
Entry into workforce Comparable Comparable
Parenting attitudes Comparable Less than Even Start
Parenting behaviors Comparable Comparable
1 Inadequate data.
Source: Auditor General staff analysis of data provided by Family Literacy Program and Even Start Program staff.
32
OFFICE OF THE AUDITOR GENERAL
(This Page Intentionally Left Blank)
33
OFFICE OF THE AUDITOR GENERAL
FINDING IV PRESCHOOL PARTICIPANTS MAKE
PROGRESS, BUT LESS THAN
EARLY CHILDHOOD BLOCK
GRANT PARTICIPANTS
The Family Literacy Program’s preschool component continues
to have an impact on increasing children’s readiness to succeed
in kindergarten. Analysis of program children’s scores on a
kindergarten readiness assessment shows that, on average, they
increased their development by three to six months in all de-velopmental
areas. Both the Family Literacy and Early Child-hood
Block Grant programs had essentially the same impact on
language skills. However, the Early Childhood Block Grant
preschool program had a greater impact on motor and prob-lem-
solving skills.
Program’s Preschool Education
Component Aims to Enhance
Kindergarten Readiness
The Family Literacy Program strives to improve the school
readiness of the 3- and 4-year-old children in the program by
providing developmentally appropriate preschool education.
To assess the degree to which this education affected children’s
progress toward school readiness, children were pre- and
posttested on the Early Success Evaluation (ESE). Chosen for
the Family Literacy Program evaluation because of its devel-opmentally
appropriate testing methodology, the ESE provides
a total score and subscores in three areas:
n Problem Solving—measures the processes by which chil-dren
acquire and use cognitive skills, such as the:
4 Ability to classify objects,
4 Understanding of numbers,
Finding IV
34
OFFICE OF THE AUDITOR GENERAL
4 Understanding of the positions of objects, and
4 Ability to order and compare objects.
n Motor Skills—measures:
4 Gross motor skills, such as balance and coordination,
and
4 Fine motor skills, such as grasp, release, and manipula-tions.
n Language—measures stages of vocabulary and sentence
structure, expression, and comprehension.
In previous Auditor General evaluations, the same test was
administered to two comparison groups of children. The devel-opment
of children in the Family Literacy Program was com-pared
to the development of children who were not in any pro-gram
and children who were in the Even Start program. Family
Literacy preschool participants had a developmental rate that
averaged four months ahead of the group that was not in any
program, and the comparison of progress between Family Lit-eracy
and Even Start children suggested the Family Literacy
children may be doing slightly better. In the current evaluation
attempts were made to administer the ESE to two comparison
groups of children: Even Start and Early Childhood Block
Grant. However, Even Start ESE scores are unavailable for use
in this evaluation because too few of the children in that pro-gram
are preschool age. This year, the ESE was administered to
a comparison group of children enrolled in the state-funded
Early Childhood Block Grant preschool program. The Early
Childhood Block Grant preschool is a stand-alone preschool
program rather than a family literacy program.
Increases in Language Skills
Were Similar Across Programs
Family Literacy children’s average language pretest scores were
12.80 on a scale of 1-29, well below the kindergarten readiness
cutoff of 20 points. They made small but statistically significant
Finding IV
35
OFFICE OF THE AUDITOR GENERAL
improvements in their language skills score, increasing it to an
average of 17.07 on the posttest. Their language skills develop-ment
increased by six months; from three years, ten months on
the pretest to four years, four months on the posttest.
Comparison of changes for Family Literacy Program partici-pants
and Early Childhood Block Grant preschool participants
on the ESE shows that both programs have essentially the same
impacts on language skills. Both groups showed improvement
from pre- to posttest.
Family Literacy Participants’
Problem-Solving and Motor Skills
Improve, but Not as Much as
Block Grant Participants’
As shown in Table 6 (see page 37), the 85 children who had
pretest scores from the start of the 1998-99 school year and
posttest results from the end of the school year made small but
statistically significant improvements toward “kindergarten
readiness” in their problem-solving and motor skills assessed
by the ESE. However, Early Childhood Block Grant preschool
participants made greater improvements in their problem-solving
and motor skills. Certain differences between the two
programs may help explain this result.
Children show modest gains in problem-solving skills—Family
Literacy preschool participants entered with an average prob-lem-
solving pre-test score of 15.4 on a scale of 1-34, well below
the 25-point cutoff for “kindergarten readiness.” They made
small but statistically significant improvement in their problem-solving
score, increasing it to an average of 20.4. Their test
scores were converted to age equivalents, which measure the
age at which the children are performing. Children in the Fam-ily
Literacy Program increased their development in problem-solving
abilities by five months; from four years, one month on
the pre-test to four years, six months on the posttest.
Average problem-solving improvement greater for Early Child-hood
Block Grant Preschool participants—Comparison of ESE
results for Family Literacy Program participants and Early
Finding IV
36
OFFICE OF THE AUDITOR GENERAL
Childhood Block Grant preschool participants shows that the
Block Grant program had a greater impact on problem-solving
skills. Block Grant preschool participants’ average problem-solving
scores were 15.9 on the pretest and 22.8 on the post-test—
an improvement close to 2 points higher than that of
Family Literacy participants. The difference between the two
groups was statistically significant. This suggests that the Early
Childhood Block Grant preschool program has a higher success
rate at improving problem-solving skills than the Family Liter-acy
preschool program component.
Children show modest gains in motor skills—Family Literacy
preschool participants began the 1998-99 school year with an
average motor skills score of 16.5 on a scale of 1-24, well below
the 22-point cutoff for “kindergarten readiness.” They made
small but statistically significant improvement in their motor
skills score, increasing it to an average of 18.5 at the end of the
program. Their development in motor skills increased by three
months; from four years, four months on the pretest to four
years, seven months on the posttest.
Average motor improvement greater for Early Childhood Block
Grant Preschool participants—The Early Childhood Block
Grant program also had a greater impact on motor skills. Block
Grant preschool participants’ average motor scores were 15.8
on the pretest and 19.8 on the posttest, an improvement 2
points higher than that of Family Literacy participants. The dif-ference
between the two groups was statistically significant.
This suggests that the Early Childhood Block Grant preschool
program also has a higher success rate at improving motor
skills than the Family Literacy preschool program component.
Certain differences between the age of participants in the two
programs may explain differences in improvement—The chil-dren
in the Family Literacy preschool component were slightly
younger on average than those in the Early Childhood Block
Grant preschool program. The average age of children in the
Family Literacy Program was 4.2 years, as compared to 4.6
years for children in the Early Childhood Block Grant Program.
In addition, there was more variation in ages among the Family
Literacy preschool participants than the Block Grant partici-pants.
Specifically, 28 percent of the Family Literacy children
were under four years old at the beginning of the 1998-99
Finding IV
37
OFFICE OF THE AUDITOR GENERAL
school year, compared to only 3 percent of the Block Grant pre-schoolers.
These differences may provide a partial explanation
for the differences in the group scores.
Future Evaluations Will Continue to
Examine the Relative Effectiveness of
the Family Literacy Program in Increasing
Kindergarten Readiness Skills
Overall, this evaluation found that Family Literacy preschool
participants improved in language, motor, problem-solving,
and total skills. However, as Table 7 (see page 38) illustrates, the
Early Childhood Block Grant preschool comparison group out-performed
the Family Literacy participants in all but one area:
language skills. In comparison, Family Literacy participants
Table 6
Family Literacy Program
Early Success Evaluation
Comparison of Family Literacy and Early Childhood Block Grant Program Participants’
Improvements in Average Test Scores
1998-99 School Year
Participants’ Average
Gain in Test Scores
Test Component
Family
Literacy
Early
Childhood
Block Grant
Language skills1 4.27 5.49
Motor skills2 1.99 3.98
Problem-solving skills3 4.98 6.91
Total improvement in average test scores3 9.90 12.48
1 The difference between the two groups is not statistically significant.
2 The difference between the two groups is statistically significant at the .01 level. That is, the probability
that the average gain occurred by chance is less than 1 in 100.
3 The difference between the two groups is statistically significant at the .05 level. That is, the probability
that the average gain occurred by chance is less than 5 in 100.
Source: Auditor General staff analysis of data provided by Family Literacy and Early Childhood Block Grant
Program staff.
Finding IV
38
OFFICE OF THE AUDITOR GENERAL
outperformed the Even Start comparison group in two areas in
the 1997 evaluation: language and total readiness. Therefore,
future evaluations conducted by the Office of the Auditor Gen-eral
will continue to examine the relative effectiveness of the
Early Childhood Block Grant Program and the Family Literacy
Program in increasing kindergarten readiness skills.
Results from the 1999 evaluation—Results from the 1999
evaluation show that on average Family Literacy preschool
participants increased their development in all developmental
areas: language, problem-solving, motor skills, and total readi-ness.
However, the Early Childhood Block Grant preschool
program had a greater impact in three of these areas: motor,
problem-solving and total readiness. Both programs had essen-tially
the same impact on language skills.
Results from the 1997 evaluation—In 1997 the Family Literacy
preschool component had an impact in increasing children’s
development in all four areas, and Family Literacy participants
outperformed the comparison group in two areas: language
and total readiness. The comparison program in the 1997
evaluation was the Even Start program, a similar Family Liter-acy
program.
Table 7
Family Literacy Program
Improvements in Child Outcomes Comparison to Even Start Program 1996-97 and
Early Childhood Block Grant Programs 1998-99
Child Outcomes 1996-97 1998-99
Language skills Greater than Even Start Comparable to Early Childhood Block Grant
Motor skills Comparable to Even Start Less than Early Childhood Block Grant
Problem-solving skills Comparable to Even Start Less than Early Childhood Block Grant
Total skills Greater than Even Start Less than Early Childhood Block Grant
Source: Auditor General staff analysis of data provided by Family Literacy Program and Even Start Program and Early
Childhood Block Grant Program staff.
39
OFFICE OF THE AUDITOR GENERAL
STATUTORY EVALUATION COMPONENTS
Pursuant to Laws 1998, Chapter 295, §8, the Office of the Audi-tor
General is required to include the following information in
the annual program evaluation.
C.1. Information on the number and characteristics of
the program participants.
The following participant information is presented in
composite from all 23 sites for the 1998-99 school year.
n Program Families—Typically, the family members
directly served by the Family Literacy Program in-clude
a mother and her three- or four-year-old child.
Occasionally a mother may have more than one
child in the preschool component (a three- and a
four-year-old, or twins). In addition, there are some
cases in which a grandparent, aunt, or father is
served by the adult education component. There are
also several cases of more than one adult from the
family being enrolled in the adult education compo-nent.
Eighty-five percent of the families in the pro-gram
have three or fewer children.
n Program Parents—The majority of adult partici-pants
are female, with only 7 percent being male.
Further, the majority of adult participants are
younger than 40 (93 percent). Participants’ ages
range from 16 to 68, with a median age of 28.
n Program Children—Thirty-six percent of the chil-dren
served are three-year-olds, and 57 percent are
four-year-olds. Sites enrolled some children who did
not meet the eligibility requirement of being 3 or 4
years old. Two percent of the children were under
three years old, and 5 percent were over four years
old.
Statutory Evaluation Components
40
OFFICE OF THE AUDITOR GENERAL
n Number of Enrolled Participants—Throughout the
1998-99 school year, 23 sites enrolled 402 families. By
statute, family literacy sites target between 10 and 20
families per site. In an attempt to maintain the mini-mum
enrollment of 10 families, sites enrolled partici-pants
continuously throughout the year as enrollment
fluctuated due to attrition. Out of 402 families, 204
completed the program.
n Family Ethnicity—The majority of families enrolled
are Hispanic (89 percent). Other ethnicities represented
include 5 percent White, non-Hispanic; 3 percent Na-tive
American; 2 percent Asian/Pacific Islanders; and
less than 1 percent African-American. “Others” or
“unknown” make up the balance. English language lit-eracy
is low, with only 15 percent of the families re-porting
English as the primary language spoken at
home.
n Family Status—Participants come from poor and un-dereducated
(have not graduated from high school)
backgrounds. Sixty-two percent of the families have in-comes
of $15,000 or less, and 29 percent of the adult
participants have less than a ninth-grade education.
n Employment Status—Though 74 percent of partici-pants
report that their primary source of family income
is wages, a vast majority (82 percent) of participants are
unemployed. Fifteen percent state that government as-sistance
is their primary source of income.
n Living Arrangement—Seventy-six percent of the par-ticipants
in the program describe their living situation
as a couple with children. An additional 14 percent live
in extended families, and 10 percent are single parents
with children.
C.2. Information on contractors and program service pro-viders.
n Thirteen Program Contractors—See Table 1 (page 5)
in the Introduction and Background for a list of these
contractors.
Statutory Evaluation Components
41
OFFICE OF THE AUDITOR GENERAL
n Two Model and Training Resource Contractors—
Mesa Unified School District’s Family Tree Project
and Pima County Adult Education served as model
and training programs. Mesa Unified School District
serves 12 sites in Arizona’s northern region (includ-ing
10 sites in Maricopa County and 2 sites in Co-conino
County), and Pima County Adult Education
serves the remaining 11 southern Arizona sites (in-cluding
4 sites in Pima County, 3 sites in Yuma
County, 2 sites in Cochise County, and 2 sites in
Maricopa County).
n Program service providers given extensive op-portunities
for family literacy training—A state-wide
Family Literacy conference was held in January
1999. This conference provided professional devel-opment
and networking opportunities for all Family
Literacy staff. The comprehensive statewide training
was a collaboration between the Arizona Depart-ment
of Education/Division of Adult Education,
model programs from Mesa and Tucson, and the
Even Start Program. In addition to this conference,
Arizona’s Family Literacy Program staff received
training in implementing a family literacy program
prior to the beginning of the 1997-98 and 1998-99
school years. The training was provided by each of
the two model programs.
The above trainings were supplemented by training ses-sions
throughout the year and on-site training con-ducted
by the two model programs.
C.3. Information on program revenues and expenditures.
Since its existence, appropriations to the Family Literacy
Program have been approximately $1 million each year.
They have been nonreverting since 1998. (See Table 2,
page 7, for information on contractors and contract
amounts.)
Statutory Evaluation Components
42
OFFICE OF THE AUDITOR GENERAL
C.4. Information on the number and characteristics of
enrollment and disenrollment.
The 23 sites served 402 families over the course of the
1998-99 school year. The sites have retained 51 percent1
of the participants for the entire school year; however,
there is great variability among these sites as far as their
success in retaining participants. Sites had retention
rates ranging from 18 to 100 percent for the 1998-99
school year. The sites that have been in existence for a
longer time tend to have better retention rates. (See Ta-ble
8, page 43.)
Of the families who exited the program prior to the end
of the school year, 126 indicated the reason they did so.
Some exits were for positive reasons, such as meeting
program goals and finding employment. (See Table 9,
page 44.)
Retention rates for the Family Literacy sites are similar
to family literacy programs across the country. As noted
in the 1997 evaluation report, the National Center for
Family Literacy (NCFL) reports that programs across the
country have a 50 percent rate of attrition, and national
studies of program effects are reported on an average of
only 10 percent of the families with matched pre- and
posttest scores.
1 Percentage is based on sites with complete posttest and exit data. Since
some participants were not enrolled in the program for the entire year,
the reported rate is deflated.
Statutory Evaluation Components
43
OFFICE OF THE AUDITOR GENERAL
Table 8
Family Literacy Program
Number and Percentage of Families Completing Programs by Site
1996-97 and 1998-99 School Years
(Unaudited)
Site
Number
Served
1996-97
Number
Completed
Percentage
Completed
Number
Served
1998-99
Number
Completed
Percentage
Completed
Cochise College 1 19 11 58%
Douglas 2 12 8 67%
Sierra Vista 2 22 9 41
Crane ESD No. 13 (2 classes) 26 18 69 32 19 59
Flagstaff USD No. 1
Killip 22 12 55 21 10 48
Leupp 3 23 NA NA 16 7 44
Isaac ESD No. 5 (2 Classes) 32 24 75 42 25 60
Literacy Volunteers of
Maricopa County 1 23 11 48
Montecito 2 8 4 50
Wesley 2 18 7 39
Mesa USD No. 4
Eisenhower 29 11 38 15 9 60
Lincoln 28 11 39 17 7 41
Longfellow 24 16 67 21 16 76
Pima County Adult Education
Liberty 18 16 89 23 7 30
Nash 15 12 80 21 13 62
Ochoa 1 16 6 38
Prince 2 17 3 18
South Tucson 2 19 11 58
Tucson USD No. 1
Lawrence 1 11 5 45
Pueblo Gardens 1 20 6 30
Tucson High School 1 18 6 33
Wakefield 1 27 13 48
Contractors with single classroom sites
Glendale ESD No. 40 2 18 7 39
Littleton ESD No. 65 2 4 4 100
Phoenix Indian Center 3 13 NA NA 21 9 43
Pinon USD No. 4 1 39 7 18
Red Mesa USD No. 27 1 29 22 76
Somerton ESD No. 11 28 11 39 16 7 44
Southwest Human Development 2 22 10 45
Tempe ESD No. 3 22   6 27 17 12 71
Total 482 224 402 204
Average 50% 51%
1 One site for the 1996-97 school year and 2 sites for the 1998-99 school year.
2 Contractor for the 1998-99 school year but not for the 1996-97 school year.
3 Completion data unavailable.
Source: Auditor General staff analysis of data provided by Family Literacy Program staff.
Statutory Evaluation Components
44
OFFICE OF THE AUDITOR GENERAL
C.5. Information on the average cost for each family in
the program.
Budgets required to operate a family literacy program
can vary depending on the number of days per week the
program operated, the program size, and the cost of
conducting the program in a community. The National
Center for Family Literacy (NCFL) estimates that one
full-time program requires an annual budget of $50,000
to $90,000 in a rural area and $75,000 to $125,000 in an
urban area (telephone conversation between NCFL staff
and Auditor General Staff, 12/9/99).
n Cost per family—The average cost per family is
$2,438 for the 1998-99 school year. Auditor General
staff calculated the cost per family using state monies
awarded to contractors for the 1998-99 school year
and the total number of families participating
throughout the school year.
n Comparable programs’ cost comparisons—The
Arizona Family Literacy Program and the federally
funded Even Start Program provide similar family
Table 9
Family Literacy Program
Number of Families Who Left the Program
and Reasons for Leaving by Percentage
1998-99 School Year
Reason Number Percentage
Met goals or completed eligible planned education 27 21.4%
Found employment that prevented further participation 27 21.4
Moved out of area 18 14.3
Conflicts or problems prevented further participation 15 12.0
Family crisis prevented further participation 14 11.1
Dropped, owing to poor attendance or incomplete participation 9 7.1
Other reason 8 6.3
Stopped participating, owing to a lack of interest 5 4.0
Switched programs 2 1.6
Reason unknown 1 0.8
Total 126 100.0%
Source: Auditor General staff analysis of data provided by Family Literacy Program staff.
Statutory Evaluation Components
45
OFFICE OF THE AUDITOR GENERAL
literacy services to families. The Even Start Program
provides family literacy services at a cost of roughly
$3,000 per family1 compared to the Arizona Family
Literacy Program that had an average cost of $2,438
during the 1998-99 school year. The federally funded
Head Start Program provided preschool services at
an average cost of $5,147 per child in fiscal year
1998.2
C.6. Information concerning progress of program par-ticipants
in achieving goals and objectives.
n Finding II (see pages 19 through 24) describes the
progress adult participants made in improving their
English language skills, attaining GEDs, and obtain-ing
employment. Adults made gains in their English
language skills, and 18 percent of participants whose
goal was to obtain their GED either successfully
completed it or were in the process of taking the test.
Additionally, at the end of the 1998-99 school year,
34 percent of the participants were employed either
full- or part-time or through a job training program
and an additional 11 percent were enrolled in a vo-cational
program.
n Finding III (see pages 25 through 31) describes the
progress adult participants have made in improving
their parenting skills. Adult participants made im-provements
in their parenting behaviors and par-enting
attitudes. Improvements in parenting behav-iors
were slightly greater than improvements in par-enting
attitudes. Most parents entered the program
with favorable parenting attitudes.
n Finding IV (see pages 33 through 38) describes the
developmental gains that the participating children
are making as a result of program participation. The
1 St. Pierre, R., B. Gamse, J. Alamprese, T. Rimdzius and F. Tao. Even
Start: Evidence from the Past and a Look to the Future. Washington, DC:
U.S. Department of Education, Planning and Evaluation Service.
2 Head Start Bureau (1999). 1999 Head Start Fact Sheet. Washington,
DC: Department of Health and Human Services.
Statutory Evaluation Components
46
OFFICE OF THE AUDITOR GENERAL
Family Literacy Program continues to have an im-pact
on increasing the kindergarten readiness of pre-school
participants.
In addition to the progress reported in Findings II
through IV, participants have achieved other outcomes.
For example, 188 participants received library cards, 32
participants earned their First Aid card, and 32 partici-pants
received CPR certification. Ten participants be-came
newly registered voters while in the program. Ad-ditionally,
3 participants became U.S. citizens while in
the program, and 39 worked on acquiring their citizen-ship.
Further, ADE reports that for fiscal year 1999, par-ticipants
completed 11,890 Community Service hours, or
approximately 17 percent of the total hours of participa-tion.
The required Community Service benchmark is
volunteer hours equal to at least 10 percent of the total
participation hours. A report documenting this activity
was submitted by the ADE to the Governor, President of
the Senate, and Speaker of the House on December 31,
1999.
C.7. Information on any long-term savings associated
with program services.
When the Family Literacy Program was continued, in-formation
on long-term savings was added as a new
element of the evaluation. For this report, evaluators
studied short- and long-term benefits provided by the
program. Although program costs were identified,
evaluators were unable to estimate short- or long-term
savings associated with the program because many of
the program’s benefits, such as improvements in child
development and improvements in adults’ skills that
can help them obtain employment, are not amenable for
translation into dollars. Additionally, other potential
program benefits, such as decreasing a child’s chances of
being retained in school, are long-term and cannot be
measured at this time.
Eventually, the program could result in increased long-term
benefits to society through helping adults complete
school, through increased wages and through tax reve-
Statutory Evaluation Components
47
OFFICE OF THE AUDITOR GENERAL
nues paid by those who obtain employment after the
program assists them in developing job-related skills. In
this evaluation, 34 percent of participants were em-ployed
at the end of the school year. Additionally, 16
percent of program participants had passed all or some
of their GED, or were attending a community college or
university at the end of the school year. Studies have
shown that the acquisition of a GED translates to as much
as a 19 percent increase in earnings over the earnings of a
high school dropout. 1
As part of future studies, evaluators will be collecting
data on long-term savings and will report this informa-tion
as appropriate in future years. OAG evaluators will
attempt to follow up on participants for several years
after they complete the Family Literacy Program. We
will attempt to collect data on higher education and
work experience after a participant attains a GED, and
on how these achievements affect income. Attempts will
also be made to collect data on former preschool partici-pants
to measure the program’s impact on school reten-tion.
C.8. Recommendations regarding program administration.
In Finding I (see pages 11 through 18), it is recommended
that:
4 ADE actively and continuously monitor sites for com-pliance
with program requirements.
4 ADE should conduct comprehensive, structured site
reviews with each site at least once a year to ensure
sites are in compliance with all statutory, program, and
contractual requirements.
1 Tyler, J., Murnane, R., and J. Willett (February 1, 1998). Estimating the
Impact of the GED on the Earnings of Young Dropouts Using a Series of
Natural Experiments. National Bureau of Economic Research Working
Paper 6391.
Statutory Evaluation Components
48
OFFICE OF THE AUDITOR GENERAL
4 ADE should take action against sites that are out of
compliance with statutory, program, or contractual re-quirements.
4 ADE should provide additional technical assistance to
sites related to test administration and reporting.
C.9. Recommendations regarding informational materials
distributed through the programs.
Family Literacy Program sites offer a variety of educa-tional
materials for families. No recommendations specific
to the informational materials are made.
Sites offer an assortment of the following informational
materials for participants: computers, reference materials,
adult and children’s books that are available for parents to
check out with their children from the site, magazines,
audio materials, parenting information, community re-source
information, GED study materials, dictionaries, sci-ence
informational materials, English language books, and
social service information.
E. Determine which program delivery models are most
effective in meeting program goals.
A variety of program delivery models were effective in
meeting program goals. The ADE contracted with a vari-ety
of providers for the Family Literacy Program. Con-tractors
include school districts, community-based organi-zations,
an adult education provider, and a college. These
contractors all provide the same four Family Literacy
components. The sites that were most effective in meeting
program goals came from each of the four types of provid-ers:
a community-based organization (Literacy Volunteers
of Maricopa County), an adult education provider (Pima
County Adult Education), a school district (Mesa Unified
School District), and a community college (Cochise Col-lege).
No program model (site) stood out as being most effective;
rather, each site met different goals with varying success,
as Table 10 (see page 50) indicates.
Statutory Evaluation Components
49
OFFICE OF THE AUDITOR GENERAL
The Montecito site, from the community-based contrac-tor’s
program model, was most effective in meeting the
Family Literacy Program goal of improving the parenting
skills of adult participants by improving their parenting
attitudes. Monticeto was also most effective in meeting the
goal of improving the literacy skills of adult participants by
improving their ability to read and write English. The
South Tucson site, from the adult education contractor’s
program model, was most effective in meeting the goal of
improving the literacy skills of adult participants by im-proving
their English listening comprehension skills. Both
the Lincoln site, from the school district contractor, and the
Douglas site, from the college contractor’s program model,
were most effective in meeting the goal of improving the
parenting skills of adult participants by improving their
parenting behaviors.
Table 10
Family Literacy Program
Sites Ranked in Comparison with Each Other by Improvement on Test Measures
1998-99 School Year
Delivery Model and Site
Comprehensive
Assessment
System of
Academic Skills
Language
Assessment
System
Parent
as a
Teacher
Behavior
Frequency
Assessment
(Monthly)
Behavior
Frequency
Assessment
(Weekly)
School district programs
Crane 15 8 18 13 7
Killip 14 13 17 10 15
Glendale 3 12 11 NA NA
Isaac 9 6 6 12 14
Littleton 2 3 2 6 4
Eisenhower NA NA 12 7 2
Longfellow 12 15 8 4 13
Lincoln NA NA 7 2 1
Community-based organization programs
Montecito 8 1 1 NA 9
Wesley 7 2 9 9 17
Phoenix Indian Center 10 7 14 3 10
Southwest Human Development 5 14 15 14 8
Adult education programs
Liberty 11 4 3 16 3
Nash 6 5 13 15 6
South Tucson 1 9 4 5 12
Prince 4 11 16 8 5
Community College programs
Sierra Vista NA NA 10 11 16
Douglas 13 10 5 1 11
Note: Leupp, Somerton, and Tempe, categorized as school district programs, had incomplete data or problems with the data. Consequently, these three
sites are not included in this table.
NA = Data not available.
Source: Auditor General staff analysis of data provided by Family Literacy sites.
Statutory Evaluation Components
OFFICE OF THE AUDITOR GENERAL
50
OFFICE OF THE AUDITOR GENERAL
AGENCY RESPONSE
1535 West Jefferson, Phoenix, Arizona 85007 l Voice: (602) 542-4391 l Fax: (602) 542-3050
State of Arizona
Department of Education
Lisa Graham Keegan
Superintendent of
Public Instruction
February 28, 2000
Ms. Debbie Davenport
Auditor General
State of Arizona
2910 North 44th Street Suite 410
Phoenix, AZ 85018
Dear Ms. Davenport:
I am responding to your agency's final evaluation of the Arizona Family Literacy Program.
A meeting to review the report draft was held in our office with members of my staff on February 7,
2000. The discussion was very beneficial.
The report contains four recommendations for program improvement and our agency addresses the
recommendations as follows:
1. The Arizona Department of Education (ADE) should actively and continuously monitor
sites each year for compliance with program requirements.
The finding of the Auditor General is agreed to and a different method of dealing with the
finding will be implemented. ADE will continue to actively monitor programs for
compliance, and is developing a comprehensive program accountability system, which
includes an electronic data collection application, to assist in this effort.
2. ADE should conduct a comprehensive, structured site review with each site at least once
a year to ensure sites are in compliance with all statutory, program and contractual
requirements.
The finding of the Auditor General is agreed to and a different method of dealing with the
finding will be implemented. ADE will continue to review programs and their sites. ADE
will determine the best design and implementation of that monitoring. An intensive,
comprehensive, structured site review each year will interfere with the provision of services
at the sites. ADE will continue to visit each site and provide technical assistance as needed.
The comprehensive accountability system will assist in ensuring program compliance.
1535 West Jefferson, Phoenix, Arizona 85007 l Voice: (602) 542-4391 l Fax: (602) 542-3050
3. ADE should take action against sites that are repeatedly out of compliance with statutory,
program and contractual requirements.
The finding of the Auditor General is agreed to and the audit recommendation will continue
to be implemented. ADE has taken appropriate action with sites repeatedly out of
compliance. Two sites that were out of compliance in Fiscal Year 1999 are no longer
operating. The program administering those sites requested a change of location to a
community with stronger support for Family Literacy.
4. ADE should provide additional technical assistance to sites related to test administration
and reporting.
The finding of the Auditor General is agreed to and the audit recommendation will be
implemented. Additional assistance by ADE has already been planned in this area.
Our agency extends our appreciation to your staff for the assistance provided in the evaluation of the
Arizona Family Literacy program.
Sincerely,
Lisa Graham Keegan
Superintendent of Public Instruction
Cc: Brian Jones
Jennifer Mabry
Karen Liersch
Lois Schneider
Other Performance Audit Reports Issued Within
the Last 12 Months
99-5 Department of Gaming
99-6 Department of Health Services—
Emergency Medical Services
99-7 Arizona Drug and Gang Policy
Council
99-8 Department of Water Resources
99-9 Department of Health Services—
Arizona State Hospital
99-10 Residential Utility Consumer
Office/Residential Utility
Consumer Board
99-11 Department of Economic Security—
Child Support Enforcement
99-12 Department of Health Services—
Division of Behavioral Health
Services
99-13 Board of Psychologist Examiners
99-14 Arizona Council for the Hearing
Impaired
99-15 Arizona Board of Dental Examiners
99-16 Department of Building and
Fire Safety
99-17 Department of Health Services’
Tobacco Education and Prevention
Program
99-18 Department of Health Services—
Bureau of Epidemiology and
Disease Control Services
99-19 Department of Health Services—
Sunset Factors
99-20 Arizona State Board of Accountancy
99-21 Department of Environmental
Quality—Aquifer Protection Permit
Program, Water Quality Assurance
Revolving Fund Program, and
Underground Storage Tank Program
99-22 Arizona Department of Transportation
A+B Bidding
00-1 Healthy Families Program
00-2 Behavioral Health Services—Inter-agency
Coordination of Services
Future Performance Audit Reports
Family Builders Pilot Program
Department of Public Safety—Aviation Division