Performance audit, Family Literacy Pilot Program

              State of Arizona
Office
of the
Auditor General
PERFORMANCE AUDIT
Report to the Arizona Legislature
By Douglas R. Norton
Auditor General
FAMILY LITERACY
PILOT PROGRAM
December 1997
Report #97-22
2910 NORTH 44th STREET • SUITE 410 • PHOENIX, ARIZONA 85018 • (602) 553-0333 • FAX (602) 553-0051
DOUGLAS R. NORTON, CPA
AUDITOR GENERAL
DEBRA K. DAVENPORT, CPA
DEPUTY AUDITOR GENERAL
STATE OF ARIZONA
OFFICE OF THE
AUDITOR GENERAL
December 11, 1997
Members of the Arizona Legislature
The Honorable Jane Dee Hull, Governor
The Honorable Lisa Graham Keegan
State Superintendent of Public Education
Department of Education
Transmitted herewith is a report of the Auditor General, A Final Evaluation of the Family Literacy Pilot Program.
The evaluation is in response to the provisions of Session Laws 1994, Ninth S.S., Ch. 1, §9.
This is the last in a series of three reports to be issued on the Family Literacy Pilot Program. The report describes
successes the Family Literacy Pilot Program has had during its first two full years of program operation. The
Program has been successful in improving adult participants’ literacy, and adults enrolled in the Program’s parenting
skills component made gains in their parenting skills, though not to the degree they made in their literacy skills.
While the children in the Program are, in general, developmentally behind where they should be for their age, the
Program keeps them from falling further behind and reduced the differences in achievement between them and their
non-at-risk peers.
A higher degree of quality and compliance with program requirements by program contractors during the 1996-97
school year in comparison with the previous year reflects hard work on the part of the contractors, improved
monitoring from the ADE, and continued technical assistance from the model programs. As a result of the successes
the Program has had, the report recommends that the Program be continued and expansion to additional sites be
considered. It is also recommended that sites serve up to 20 families, an increase from the current maximum of 15,
in order to reduce the costs per family. If the Program is continued, it is recommended that model program sites also
be continued in order to assist with ongoing programs’ training and technical assistance needs. In addition, the
report also recommends that the ADE continue to monitor and report annually through 2002 to the Legislature on
the progress of Family Literacy participants.
As outlined in its response, the Department of Education agrees with all of the findings and recommendations.
My staff and I will be pleased to discuss or clarify items in the report.
This report will be released to the public on December 12, 1997.
Sincerely,
Douglas R. Norton
Auditor General
Enclosure
i
SUMMARY
The Office of the Auditor General has completed the final in a series of three annual evalua-tions
of the Family Literacy Pilot Program. This evaluation was conducted pursuant to the
provisions of Session Laws 1994, Ninth S.S., Ch. 1, §9. This third-year evaluation report
provides information regarding the Program’s effectiveness in achieving its goals.
The Legislature established the Family Literacy Pilot Program through legislation known as
the Children and Family Stability Act of 1994. The Program is administered by the State
Board of Education through the Arizona Department of Education (ADE) Division of Adult
Education. ADE contracts with local school districts and other community and educational
organizations to provide program services.
The Family Literacy Program’s intent is to increase the basic academic and literacy skills of
undereducated parents of preschool children in order to break the intergenerational cycle of
poverty and illiteracy. The Program is based on the belief that children’s early learning is
greatly influenced by their parents. Therefore, parents must develop and value their own
literacy skills in order to support their children’s educational success. More important,
parents are their children’s first and best teachers.
Family literacy programs incorporate adult, child, and parent education in a manner that is
theoretically more effective than programs that focus exclusively on adults or exclusively on
children. Arizona’s program uses a model that integrates components recommended by the
National Center for Family Literacy. Families receive services in a classroom setting for at
least 15 hours a week, and programs generally operate on regular school year calendars. The
Program’s four main components are: 1) adult literacy instruction, 2) parent and child to-gether
(PACT), 3) parent education discussion and support, and 4) early childhood educa-tion
for children ages 3 to 4 years.
Adult Family Literacy
Participants Show Gains
in Literacy Skills
(See pages 9 through 14)
The Program is successful in improving adult participants’ literacy. Standard test scores
increased both for adults enrolled in English for Speakers of Other Languages (ESOL) and
for those enrolled in adult basic education/general equivalency degree preparation
(ABE/GED). Twenty-six percent of the participants either completed their GED through the
Program or were in the process of taking the examinations at the end of the school years.
Although many participants are focusing on raising families, and therefore not seeking
ii
employment, three-fourths of those participants seeking employment have entered the
workforce.
Although it is too early to measure long-term effects, the positive adult education outcomes
offer the possibility that the Program may have the intended impact of breaking the
intergenerational cycle of poverty and illiteracy. As the parents’ educational and literacy
skills improve, it is expected that their children will do better in school. Not only should the
preschool children in the Family Literacy Program do better, but their older and younger
siblings should also do better.
Family Literacy Pilot Program
Participants Show Slight Improvements
in Parenting Behaviors
(See pages 15 through 19)
The model used by Arizona’s Family Literacy Program recognizes the importance of
parenting skills and focuses on developing these skills through the parent and child together
(PACT) and parenting group discussion components.
Most parents entered the Program with relatively good attitudes about parenting. Adults
enrolled in the parenting skills component of the Family Literacy Program made gains in
their skills, though not to the degree that they made gains in their literacy skills. However,
the parents who had the poorest parenting attitudes when they entered the Program
showed significant improvement in their parenting attitudes. In addition, parents’
comments regarding their parenting behaviors indicate that their behaviors may have
changed more than the measured changes in their attitudes would indicate.
Family Literacy Preschool
Participants Making Progress
(See pages 21 through 23)
A major objective of the Family Literacy Pilot Program is to improve the school readiness of
the 3- and 4-year-old children in the Program by providing developmentally appropriate
preschool education. A kindergarten readiness assessment suggests that children who
participate in the Program, as a result of their participation, show a developmental rate that
is an average of four months ahead of nonparticipating children. While children are still
generally behind where they should be for their age, the Program keeps them from falling
further behind and reduces the differences between them and their non-at-risk peers. As a
result of the Program, children are reducing the degree to which they are behind and are
more likely to start school prepared to learn.
iii
Program Implementation and
Monitoring Improves in Second
Full Year of Operation
(See pages 25 through 31)
Most contractors operating sites for the Family Literacy Program have made progress in
offering quality programs since last year’s report. Site evaluations showed that 20 of the 23
sites that were evaluated made progress in meeting statutory requirements and correcting
previous quality-related problems in such areas as integrating components, collaborating
with other providers, providing childcare, and maintaining enrollments at required levels.
The higher degree of quality and compliance reflects hard work on the part of the sites,
improved monitoring from ADE, and continued technical assistance from the model pro-grams.
However, three of the sites, Pinon, Leupp, and Phoenix Indian Center, continue to have
serious implementation problems. These sites have made staff changes but continue to
struggle with quality and enrollment. ADE has made efforts to improve the programs and
all three have developed plans to implement changes for improvement. These programs
have been funded for the 1997-98 school year on a month-to-month contract instead of the
quarterly payment plan used for the other programs.
If the Legislature continues the Program, it is recommended that ADE continue monitoring
efforts to ensure program quality remains. And, if the Program is expanded to additional
sites, it is recommended that model program activities designed to provide technical assis-tance
are continued to ensure that new sites have the assistance they need to succeed.
Statutory Evaluation Components
(See pages 33 through 46)
Laws 1994, Ninth S.S., Ch.1, §9 requires the Office of the Auditor General to report whether
program participants have achieved goals and objectives, and make recommendations
regarding program expansion.
The Family Literacy Pilot Program positively impacts the families who choose to enroll into
and stay in the Program. The Program appears to be successful in improving the literacy
rates of the adult participants, and has some positive effects on their parenting behaviors. In
addition, preschoolers in the Program are making gains beyond what is predicted for them.
Based on positive program outcomes and the achievement of program goals and objectives,
it is recommended that the Program be continued and expansion to additional sites consid-ered.
Positive impacts were found even though there were serious problems during the first
two years of program implementation. As noted in our second evaluation of the Program
last year (Report No. 96-20), only 7 of 26 programs complied with all program criteria. The
iv
second-year evaluation warned that the problems with implementation threatened the
Program’s potential for success. The fact that positive impacts were found despite the im-plementation
problems suggests that greater impacts may be found in the future if the
implementation problems continue to be addressed.
However, if the Program is continued or expanded, because Arizona’s family literacy model
cannot meet the needs of all adults and children in need of literacy services, any additional
monies should be directed to areas with the greatest opportunity to make an impact. Monies
should be distributed through a request for proposal process requiring potential contractors
to demonstrate high rates of economic and educational disadvantage in their service area
and a workable plan to attract and retain the target population.
It is also recommended that model program sites be continued in order to assist with ongo-ing
programs’ training and technical assistance needs as they experience staff turnover, and
new sites as they begin to provide services. Model programs should provide technical
assistance, training, and basic monitoring in cooperation with ADE staff.
v
Table of Contents
Page
Introduction and Background.............................................................. 1
Finding I: Adult Family Literacy
Participants Show Gains in
Literacy Skills .................................................................................... 9
Background................................................................................................................... 9
ESOL Parents Improve Their
English Language Skills .............................................................................................. 10
ABE/GED Parents Show Improvements ................................................................. 11
Some Participants Received GEDs;
a Few Attended College.............................................................................................. 12
Family Literacy Participants
Enter the Workforce..................................................................................................... 13
Participants Self-Report
Positive Impacts............................................................................................................ 13
Family Literacy Results
Similar to Even Start Results....................................................................................... 13
Conclusions................................................................................................................... 14
Recommendations........................................................................................................ 14
Finding II: Family Literacy Pilot Program
Participants Show Slight Improvements
in Parenting Behaviors..................................................................... 15
Background................................................................................................................... 15
Parents Enter Program with
Good Parenting Attitudes........................................................................................... 16
vi
Table of Contents (cont’d)
Page
Finding II: (cont’d)
Parents Who Enter with
Unfavorable Parenting
Attitudes Make Large Gains....................................................................................... 16
Most Parents Show Minimal Gains
in Parenting Attitudes As
Measured by the PAAT............................................................................................... 17
Behavior Frequency
Assessment Shows Little
Overall Change in Behaviors...................................................................................... 17
Family Literacy and Even
Start Show Same Impacts............................................................................................ 17
Parents Report Improved
Parenting Behaviors..................................................................................................... 18
Conclusions................................................................................................................... 19
Finding III: Family Literacy Preschool
Participants Making Progress......................................................... 21
Background................................................................................................................... 21
Data Analysis Reveals
Program Has Impact
on Children’s Development........................................................................................ 21
AESOP May Fail to
Measure Some Growth................................................................................................ 23
Comparison Reveals Family Literacy
Children May Be Doing Better
Than Even Start Children............................................................................................ 23
Conclusion..................................................................................................................... 23
vii
Table of Contents (cont’d)
Page
Finding IV: Program Implementation and
Monitoring Improves in Second
Full Year of Operation ...................................................................... 25
Background................................................................................................................... 25
Sites Improve Compliance with
Statutory and Contractual Guidelines....................................................................... 26
ADE Improves Monitoring......................................................................................... 27
Model Sites Provide Valuable
Technical Assistance..................................................................................................... 27
Quality Improves at
Family Literacy Sites.................................................................................................... 30
Recommendations........................................................................................................ 31
Statutory Evaluation Components...................................................... 33
Agency Response
Appendix A.............................................................................................. a-i
Appendix B.............................................................................................. b-i
Appendix C.............................................................................................. c-i
viii
Table of Contents (cont’d)
Page
Tables
Table 1 Family Literacy Pilot Program
Contractors and Contract Amounts
Years Ended or Ending June 30, 1995 through 1998
(Unaudited)................................................................................................ 5
Table 2 Family Literacy Pilot Program
Percentage of Adults Whose Literacy Level Improved
As Measured by Pre- to Posttesting on the
Language Assessment System
1996-97 School Year................................................................................... 11
Table 3 Family Literacy Pilot Program
Adult Pre- and Posttest Scores in Grade Equivalents
1995-96 and 1996-97 School Years........................................................... 12
Table 4 Family Literacy Pilot Program
Pre- to Post-Program Changes in Parenting Behaviors
1995-96 and 1996-97 School Years........................................................... 18
Table 5 Family Literacy Pilot Program
Activity Evaluation System for Observing Preschoolers
Median Score in Months by Age Group
1995-96 and 1996-97 School Years........................................................... 22
Table 6 Family Literacy Pilot Program
Instances of Noncompliance by Site and Requirement
Years Ended June 30, 1996 and 1997....................................................... 28
Table 7 Family Literacy Pilot Program
Number and Percentage of Families
Who Completed Programs per Site
1995-96 and 1996-97 School Years
(Unaudited)................................................................................................ 37
ix
Table of Contents (concl’d)
Page
Tables (concl’d)
Table 8 Family Literacy Pilot Program
State Cost per Family per Site
Years Ended June 30, 1996 and 1997
(Unaudited)................................................................................................ 39
Table 9 Family Literacy Pilot Program
Cost of Comparable Programs
1995-96 and 1996-97 School Years........................................................... 41
Table 10 Family Literacy Pilot Program
English for Speakers of Other Languages
Participants Matched Pre- to Post-t-test
Language Assessment Scales (LAS) and
Comprehensive Adult Student Assessment System (CASAS) Tests
1995-96 and 1996-97 School Years........................................................... c-i
Table 11 Family Literacy Pilot Program
Adult Basic Education Participants Matched Pre- to Post-t-test
Test of Adult Basic Education (TABE)
1995-96 and 1996-97 School Years........................................................... c-iii
Table 12 Family Literacy Pilot Program
Parent As a Teacher Inventory
Matched Pre- to Post-t-test
1995-96 and 1996-97 School Years........................................................... c-v
Table 13 Family Literacy Pilot Program
Preschoolers’ Pre- to Posttest Gains in Performance Age
1995-96 and 1996-97 School Years........................................................... c-vii
x
(This Page Intentionally Left Blank)
1
INTRODUCTION AND BACKGROUND
The Office of the Auditor General has completed the final in a series of three annual evalua-tions
of the Family Literacy Pilot Program. This evaluation was conducted pursuant to the
provisions of Session Laws 1994, Ninth S.S., Ch. 1, §9. This final evaluation report provides
information regarding the Program’s effectiveness in achieving its goals and recommenda-tions
regarding it.
The Family Literacy Program:
What It Does
The Legislature established the Family Literacy Pilot Program in 1994 with legislation
known as the Children and Families Stability Act. The Program’s intent is to increase the
basic academic and literacy skills of undereducated parents of preschool children in order to
break the intergenerational cycle of poverty and illiteracy. The Program is based on the
premise that in order to increase literacy and reduce poverty among the current generation,
the educational skills of parents and children must increase.
The National Center for Children in Poverty reports that children whose parents lack a high
school diploma are more than twice as likely to live in poverty than children whose parents
are high school graduates. When parents improve their literacy skills, they can improve their
families’ lifestyles by improving such things as employment options, consumer and finan-cial
skills, and knowledge of health and personal safety techniques (improved knowledge
and use of medicines, instructions, first aid, and emergencies). In terms of children, other
researchers, such as the Perry Preschool researchers, have found that preschool participation
for at-risk children leads to positive long-term outcomes, such as increased levels of high
school completion, better employment, higher incomes, a reduced dependency on welfare,
and lower rates of involvement in the criminal justice system.
The Arizona Family Literacy Program is based on the belief that children’s early learning is
greatly influenced by their parents. Therefore, parents must develop and value their own
literacy skills in order to support their children’s educational success. More important,
parents are their children’s first and best teachers. The skills that they bring to the task are
critical to their children’s healthy development and acquisition of literacy.
Not only is the Family Literacy model designed to help the entire family, research suggests
the family members who are direct participants in the Program are not the only ones to
benefit. For example, older siblings begin to perform better in school after their parents
enroll in the Program. Additionally, research shows that family literacy means earned in-come,
a healthy family, better use of community resources, reduction in school failure, and
reduction in need for special services for children.
2
Participants
Eligible parents 1) have a three- or four-year-old child; 2) lack sufficient mastery of basic
educational or basic English language skills needed to function in society, or lack a high school
diploma or its equivalent; and 3) are U.S. or legal residents, or are otherwise lawfully present
in this country. Programs are located in areas with high rates of unemployment and low rates
of literacy. Families enroll in the Program through a variety of local program recruitment
efforts.
Components
Family literacy programs incorporate adult, child, and parent education in a manner that is
theoretically more effective than programs that focus exclusively on adults or exclusively on
children. Arizona’s program uses a model that integrates components recommended by the
National Center for Family Literacy. Families receive services in a classroom setting for at
least 15 hours a week, and programs generally operate on regular school year calendars. The
Program’s main components are as follows:
n Adult literacy instruction—Adult literacy education takes two basic forms: 1) adult
basic education (ABE) and preparation for the general education diploma (GED), or 2)
instruction in English for Speakers of Other Languages (ESOL). Parents can receive one
or both components depending on their needs.
n Parent and Child Together (PACT)—Through situations in which the family plays
together or is involved in more structured activities, PACT tries to show parents how to
teach and communicate with their children using positive parenting skills that are trans-ferable
to the home.
n Parent education discussion and support groups—Along with PACT, these compo-nents
of the Program attempt to provide parents with opportunities that will help them
learn how to best meet their children’s developmental needs. Discussion and support
groups make it easier for parents to discuss issues or problems surrounding parenting.
Participants learn from the experiences of their peers, receive encouragement from the
group, and practice collective problem solving.
n Early childhood education—A major component of the model used by the Family
Literacy Pilot Program is the provision of early childhood education to the young chil-dren
in the Program in order to improve their school readiness and increase their
chances of future academic success. Each program site is to provide participating chil-dren
with a developmentally appropriate preschool to improve motor, language, and
problem-solving skills.
3
When the Family Literacy Program is working effectively, it can produce results like the
following, which are quotes from actual program participants:
I made the effort and changed my life to set an example for my daughters. Now I don’t get
afraid when people speak to me in English, because thanks to this program I can understand
what they are saying.
(Before Family Literacy) my daughter would spend hours in front of the TV always bored.
(Now) she has learned more skills and to share with her friends, and has improved her lan-guage
and she learned words, songs, colors, how to use the computer, counting, to tell stories,
to read books, how to brush her teeth, and the classroom rules, the parts of her body. . . she
doesn’t feel embarrassed with her friends and she is more independent about how to make her
own decisions.
How the Program Is Operated
The Program is administered by the State Board of Education through the Division of Adult
Education, Arizona Department of Education (ADE). ADE contracts with other organiza-tions
to provide services. The Legislature mandated that contractors provide the following
basic services:
n Identify and recruit eligible parents and children.
n Screen and prepare parents and children for participation in the Program.
n Serve at least 10 but no more than 15 parents with children eligible for the Program.
n Provide instructional programs that promote academic and literacy skills and equip
parents to provide needed support for their children’s educational growth and success.
To encourage families to fully participate, contractors must also have a plan for providing
food services, childcare, and transportation for participants.
The Program is flexible in the length of time families are served. Families must have an age-eligible
child and the parent(s) must have literacy goals that can be met through the Pro-gram.
While most families are only in the Program one year, some families return for a
second or third year.
Contractors for 1996-97 include 9 school districts, 2 community-based organizations, 1
college, and 1 adult education provider. In all, these contractors operate 23 sites, 18 of which
4
serve metropolitan Phoenix, Tucson, or Yuma. The remaining 5 sites serve Arizona’s rural
population, with 2 sites in Coconino County and 1 site each in Apache, Cochise, and Navajo
Counties. Two model programs, Mesa Family Tree and Pima County Adult Education,
provide technical assistance to all sites and training to family literacy staff.
Appropriations for the Family Literacy Pilot Program were $975,000 and $1,000,600 for
school years 1994-95 and 1995-96, respectively. The Legislature appropriated $1 million for
both the 1996-97 and 1997-98 school years. Table 1 (see page 5) shows the contractors and
state Family Literacy Pilot Program grant amounts for the 1994-95, 1995-96, 1996-97, and
1997-98 school years.
Although funding began in the 1994-95 school year, contractors generally did not provide
direct services to families until the 1995-96 school year, because implementation was de-layed
due to the lengthy grant award process. Grant amounts for the 1994-95 school year
reflect that most contractors provided only minimal services. Two rural sites that provided
services during the 1995-96 school year did not have contracts renewed for 1996-97 because
they failed to meet their contractual obligations.
As described in this Office’s first- and second-year reports, ADE had carryover monies from
the 1994-95 school year that allowed the agency to support programs beyond the appro-priations
level in the 1995-96 school year. However, since then ADE has had to reduce the
level of funding for each site, and no new sites have been funded since the 1994-95 school
year.
Follow-Up of Previous
Evaluation Reports
Program Model
The first and second evaluation reports (Report No. 95-20—December 1995, and Report No.
96-20—December 1996), focused on program implementation. In the first year, it was re-ported
that the program model used by the Family Literacy Pilot Program followed recom-mended
standards. The second year it was reported that many sites failed to comply with
guidelines and consequently, the model described in the first report was in some cases
poorly implemented. Failure to adhere to the program model resulted in two sites not
having their contracts renewed. In this third and final report, compliance improvements at
most sites are noted (see Finding IV, pages 25 through 31). However, three sites continue to
have implementation problems and ADE is closely monitoring them to ensure they meet the
program guidelines. Additionally, providers chose to close two sites and use these monies
for their other sites.
5
Table 1
Family Literacy Pilot Program
Contractors and Contract Amounts
Years Ended or Ending June 30, 1995 through 1998
(Unaudited)
Contractor 1995 1996 1997 1998
Apache County
Red Mesa USD No. 27 $ 73,178 $ 87,930 $ 66,387 $ 66,387
Cochise County
Cochise Community College 17,746 44,265 33,420 33,419
Council for Family Concern 18,875 43,500
Coconino County
Flagstaff USD No. 1 (2 sites) 157,794 74,888 56,541 56,540
Maricopa County
Isaac ESD No. 5 (2 classes) 19,927 87,997 66,438 66,438
Literacy Volunteers of Maricopa County 16,420 40,064 30,248 30,248
Mesa USD No. 4 (3 sites) 56,341 110,543 83,460 83,460
Phoenix Indian Center 20,712 43,234 32,642 32,642
Tempe ESD No. 3 50,000 44,000 33,220 33,220
Navajo County
Pinon USD No. 4 83,676 121,222 91,523 91,523
Pima County
Pima County Adult Education (3 sites) 137,531 131,984 99,648 99,648
Tucson USD No. 1 a 80,000 220,000 166,100 165,968
Yavapai County
Chino Valley USD No. 51 6,942 24,549
Yuma County
Crane ESD No. 13 (morning session) 21,400 37,632 28,412 28,412
Crane ESD No. 13 (afternoon session) 21,400 37,632 28,412 28,412
Somerton ESD No. 11 30,000 61,600 46,508 46,508
Model Programs
Mesa USD No. 4 52,045 61,324 69,686 69,688
Pima County Adult Education 69,915 61,560 67,355 67,355
Total $933,902 $1,333,924 $1,000,000 $999,868
a Tucson USD No. 1 had five sites in 1995 and 1996 and four in 1997, and has two in 1998.
Source: Auditor General staff analysis of data provided by the Arizona Department of Education.
6
Budgeting Problems
In the first year it was reported that ADE’s poor budgeting resulted in a money shortage for
second-year services. In the second-year report, the negative impact of the problems caused
by the first-year budgeting error was discussed. While this issue has not been focused on in
this final report, there remains a concern that low funding levels may continue to impact the
quality of some programs. Funding is discussed in the Statutory Annual Evaluation Com-ponents.
Inadequate Monitoring
The first-year report identified inadequate program monitoring and oversight by the ADE.
In the second year the problem continued, with ADE failing to provide adequate monitoring
for compliance with the program model. In Finding IV (see pages 25 through 31) of the third
report, it is reported that ADE made great improvements in program monitoring and pro-viding
technical assistance during the 1996-97 school year.
Scope and Methodology
Methods used in this evaluation include pretesting to posttesting analysis of nationally
standardized tests designed to measure adult basic education skills, English language skills,
and parenting skills. Pre- to post-analysis of parents’ reported behaviors specific to parent-ing
and pre- to post-analysis of children’s development was also performed. All pre- to post-analysis
is for both the 1995-96 and 1996-97 school years. Additional methods include analy-sis
of reported outcomes for families in the Program as reported in exit forms for both pro-gram
years. During the 1996-97 school year, 2 structured site observations were conducted
at 21 of the 23 sites and 1 structured observation at the remaining 2 sites. Program trainings
were observed, and site documents were reviewed. In addition, 4 families were interviewed
2 times over the course of the Program and participants’ journals were reviewed to deter-mine
what impact, if any, program participation had on the families’ lives.
The design used in this evaluation of the Family Literacy Pilot Program is quasi-experimental,
with pre- and post-assessment of participants and a comparison group of
families served by Even Start. Even Start is the federally funded family literacy program,
with sites funded at levels similar to those for the family literacy sites. The Even Start Pro-gram
is a national leader in providing family literacy programs. Attempts to recruit adult
education and at-risk preschool participants into the comparison group, which would allow
for an assessment of how effective this family literacy program is in contrast to stand-alone
programs, were unsuccessful because of program staff’s concerns about the testing and
possible results. Therefore, the only comparison made was to the Even Start Program, which
allows for an assessment of how this Family Literacy Pilot Program compares to another
family literacy program.
7
This third and final report focuses on the Family Literacy Pilot Program’s effectiveness in:
n Improving the basic literacy skills and educational and employment achievements of the
adults in the Program;
n Improving the parenting skills of the adults in the Program; and
n Improving the school readiness skills of the three- and four-year-old preschoolers in the
Program.
The report also focuses on improvements in program implementation issues.
In the Statutory Evaluation Component, evaluation requirements as defined by Laws 1994,
Ninth S.S., Ch. 1, §9 are addressed. Included in these requirements are recommendations
regarding program administration and expansion. Based on the conclusions regarding
positive program effects and noted improvements in the program administration and im-plementation,
a recommendation is made to continue, and possibly gradually expand, the
Program.
This report also contains three technical appendices. Appendix A (see pages a-i through a-iv),
provides a detailed description of the instruments used to evaluate the program out-comes.
Appendix B (see page b-i), describes the testing protocols. Appendix C (see pages c-i
through c-vii), provides summary statistical tables presenting the results of statistical analy-sis
of the data used in this evaluation.
The Auditor General and staff express appreciation to the Superintendent of Public Instruc-tion,
staff of the Arizona Department of Education’s Adult Education Division, and the
Family Literacy and Even Start Programs’ staff and families for their cooperation, assistance,
and hospitality as auditors visited their classrooms throughout the three years of this
evaluation.
8
(This Page Intentionally Left Blank)
9
FINDING I
ADULT FAMILY LITERACY PARTICIPANTS
SHOW GAINS IN LITERACY SKILLS
Nearly all of the participants in the Family Literacy Program showed improvements. Adults
enrolled in the adult literacy component made gains in their literacy skills. Standard test
scores increased both for adults enrolled in English for Speakers of Other Languages (ESOL)
and for those enrolled in adult basic education/general equivalency degree preparation
(ABE/GED). Twenty-six percent of the participants either completed their GED through the
Program or were in the process of taking the examinations at the end of the school years.
Although many participants are focusing on raising families and, therefore, are not seeking
employment, three-fourths of those participants seeking employment have entered the
workforce. The Family Literacy Program’s success is equal to that of Even Start, a
comparable program that is federally funded and operated.
Background
Family Literacy’s adult education component is founded on the theory that by improving
parents’ literacy skills, their children will do better in school and parents will have better
economic and family skills. To determine if Family Literacy participants made significant
improvements, participants were tested and their results were compared to those
participants in a similar federal program. ESOL and ABE/GED participants from two
separate school years (1995-96 and 1996-97) took standardized tests before entering the
Family Literacy Program and at the end of each school year. Analysis of the scores pre- and
post-program was used to determine if the Family Literacy Pilot Program was effective in
improving their literacy skills.
English for Speakers of Other Languages participants were pre- and posttested using the
Language Assessment System to assess their English language reading and writing skills and
the Comprehensive Assessment System of Academic Skills to assess their English language
listening comprehension. ABE/GED participants were pre- and posttested using the Tests of
Adult Basic Education for mathematics, language, reading, and spelling. A comparison was
made between parents participating in Family Literacy and parents participating in Even
Start, the federally funded family literacy program. Even Start participants were pre- and
posttested using the same assessments to determine if there was any difference in the
effectiveness of the two programs.
10
A description of the tests is provided in Appendix A (see pages a-i through a-iv). The testing
protocol is described in Appendix B (see page b-i).
ESOL Parents Improve Their
English Language Skills
The 342 adults who participated in the ESOL component in either the 1995-96 or 1996-97
school year made modest gains in English language skills (for complete results see Table 10,
page c-i in Appendix C). They made statistically significant, though modest, gains on all
four standardized tests for both years. Their improvements in reading and writing were
greater than their gains in ability to understand spoken English.
English language reading and writing skills improve—Participants’ abilities in their English
reading and writing skills, as measured by reading, writing, and the combined
reading/writing score on the Language Assessment System, increased in both years. All of
the gains were statistically significant at the .001 level, meaning there is a less than 1 in 1,000
chance of the average gain occurring by chance.
In addition to being statistically significant, the Language Assessment System gains suggest
real improvements in the participants’ abilities to read and write English. Language
Assessment System reading and writing scores can range from a low of 0 to a high of 100.
On this 100-point scale, the 1996-97 participants’ median reading gain was 14 points, and the
median writing gain was 18 points. The developers of the test group scores into five
categories of competence so that the numeric scores can be understood. The five categories
are low beginner (0 to 20), high beginner (21 to 40), low intermediate (41 to 60), high
intermediate (61 to 80), and competent (81 to 100). Generally, Family Literacy participants
advanced to higher groups. More specifically: first-year and second-year participants
moved their average reading scores from low intermediate to high intermediate and their
writing scores from high beginner to low intermediate.
As seen in Table 2, page 11, the majority of participants made gains that moved them into at
least one higher level or category of literacy as defined by the developers.
Gains in listening comprehension skills more modest— While both first- and second-year
participants made gains in English listening comprehension skills, the gains were more
modest than for reading and speaking skills. In both years the participants’ scores moved up
minimally but stayed in the category the Comprehensive Assessment System of Academic
Skills publishers call “low literacy.” During the second year, 78 percent of the ESOL
participants improved their listening scores, compared with 65 percent the first year,
although these participants are still in the “low literacy” range.
11
Table 2
Family Literacy Pilot Program
Percentage of Adults Whose Literacy Level Improved
As Measured by Pre- to Posttesting on the
Language Assessment System
1996-97 School Year
Literacy Level Reading Writing
Improved 57% 66%
Stayed the same 40 32
Decreased 3 2
Source: Auditor General staff analysis of data provided by Family Literacy Program staff.
ABE/GED Parents Show Improvements
Like the ESOL participants, the 159 adults who participated in the ABE/GED component
and were both pre- and posttested in either the 1995-96 or 1996-97 school year made gains.
They recorded gains in all four categories tested in the Tests of Adult Basic Education—
applied math, language, reading, and spelling abilities (see Table 11, pages c-iii through c-iv,
Appendix C). The gains in most of the pre- and posttests were statistically significant at the
.001 level or less, meaning that they would have less than a 1 in 1,000 likelihood of occurring
by chance.
The extent of these gains may be more understandable if the scores are converted to grade
levels (see Table 3, page 12). Grade equivalent scores have a potential range of 0
(kindergarten) to 12.9 (9 months into 12th grade). For example, 1995-96 participants began
with an applied math skill level equivalent to an ending third grader. They completed the
year scoring at the equivalent of an average student 6 months into sixth grade. In language
skills, the first-year group started the year equivalent to 9 months into second grade and
finished the year equivalent to 4 months into fifth grade.
12
Table 3
Family Literacy Pilot Program
Adult Pre- and Posttest Scores in Grade Equivalents1
1995-96 and 1996-97 School Years
1995-96 1996-97
Test component Pretest Posttest Pretest Posttest
Applied math 3-9 6-6 4-8 6-4
Language 2-9 5-4 5-2 6-8
Reading 4-4 6-3 5-2 7-3
Spelling 6-5 7-5 7-6 9-0
1 A score reported as a grade equivalent means it is the score typical for a student in that grade. For example,
3-9 means it is the typical score for a student in the ninth month of third grade.
Source: Auditor General staff analysis of data provided by Family Literacy Program staff.
As with the ESOL program, gains were widespread rather than concentrated among a few
participants, and more participants experienced gains in the second year than in the first
year. The percentage of second-year ABE/GED participants who had gains ranged from 64
percent for language to 79 percent for reading. First-year participants who had gains ranged
from 54 percent for spelling to 85 percent for applied mathematics.
Some Participants Received GEDs;
a Few Attended College
Of the 351 parents who participated in the adult education component of the Family Liter-acy
Program in the 1995-96 and 1996-97 school years, more than 40 have received their GED,
and more than 50 were reportedly in the process of taking the GED tests at the end of the
school year but had not yet taken or passed all of them. Another 201 participants were
reported to be continuing to work toward their GED at the end of the school years.
A small number of Family Literacy participants have continued on to college. Four
participants have been accepted to attend college. Since there is little follow-up of
participants after they leave the Program, it is possible that more adult participants have
continued their education after completing their GED.
13
Family Literacy Participants
Enter the Workforce
Of those Family Literacy participants who sought employment at the end of the Program, 76
percent were employed. One of the Program’s goals is to increase participants’ self-sufficiency,
and employment is one measure of self-sufficiency.
At the end of the 1995-96 school year, 30 percent of the participants were employed either
full- or part-time or through a job training program. An additional 14 percent were enrolled
in a vocational program, and 15 percent were looking for work. Almost identical
employment outcomes were present at the end of the second year. For both years,
approximately 40 percent of the participants ended the school year unemployed but
currently not seeking employment.
Participants Self-Report
Positive Impacts
The statistical results are further substantiated with qualitative information on program
impacts. Participants’ statements report a positive impact from the adult education compo-nent:
“I met new friends and started learning English. It was good for me because I can speak Eng-lish
with the doctor. . . Also I know how to use the computer . . .it is good for me, because this
can help me to get a job.”
“My life is clear now and my future is also. They are possible because of the knowledge I
gained in my English class. My opinion has a voice and my thoughts carry weight. I’m a per-son
with a good future.”
“And my wish was to learn English and to be more prepared to get a better job. . . I have better
communication when I go to store, bank, or hospital.”
“I feel more sure of myself. I am starting a road but still lack many things to achieve my
dreams, to learn better English, and to graduate from the U of A.”
Family Literacy Results
Similar to Even Start Results
Even Start sites across Arizona were assessed using the same tests as Family Literacy sites.
No significant differences in gains were found between the Family Literacy and Even Start
participants on the Language Assessment System, Comprehensive Assessment System of
Academic Skills, or Test of Adult Basic Education, or on the GED and employment
14
outcomes. This suggests that the state program is having about the same level of success as
the federal program.
Conclusions
The Program is successful in improving the adult participants’ literacy rates. Although it is
too early to measure long-term effects, the positive adult education outcomes offer the
possibility that the Program may have the intended impact of breaking the intergenerational
cycle of poverty and illiteracy. As the parents’ educational and literacy skills improve, it is
expected that their children will do better in school. Not only should the preschool children
in the Family Literacy Program do better, but their older and younger siblings should also
do better.
Recommendations
If the Program is continued, the Arizona Department of Education should monitor the
progress of adult participants after leaving the Program for 1) GED completion; 2) college
acceptance and attendance; and 3) employment. The ADE can accomplish this by requiring
sites to follow up on participants at six-month intervals for the first two years after they
leave the Program and reporting this information to the ADE.
15
FINDING II
FAMILY LITERACY PILOT PROGRAM
PARTICIPANTS SHOW SLIGHT IMPROVEMENTS
IN PARENTING BEHAVIORS
Adults enrolled in the parenting skills component of the Family Literacy Program made
improvements in their attitudes, though not to the degree that they made gains in their
literacy skills. Most parents in both years entered the Program with relatively good
parenting attitudes and did not show great improvements. However, large improvements in
parenting attitudes were found in the parents who had the poorest parenting attitudes when
they entered the Program. In addition, parents’ comments regarding their parenting
behaviors indicate that their behaviors may have changed more than the measured changes
in their parenting attitudes would indicate. A comparison with gains made by participants
in the federally funded Even Start Program showed that both programs had essentially the
same impacts on participants’ skills.
Background
The model used by Arizona’s Family Literacy Program recognizes the importance of
parenting skills and focuses on developing these skills through the parent and child together
(PACT) and parenting group discussion components. These components are intended to
help parents help their children learn through play and more structured activities and
provide parents with opportunities to learn how to best meet the developmental needs of
their young children.
To assess the extent to which the Family Literacy Pilot Program impacted adults’ parenting
skills, participants were pre- and posttested on two instruments. See Appendix A (pages a-i
through a-iv), for additional information regarding the assessments and Appendix B (page
b-i), for a discussion of the testing protocols.
The first instrument is the Parent as a Teacher Inventory (PAAT). The PAAT identifies
favorable qualities and realms in which personal growth is needed in regard to parenting
children ages 3 to 9. PAAT items include a variety of statements on what parents want or
expect of their child, how they interact with the child, and what actions are taken in
response to specific child behavior. The responses are grouped into five areas with subscores
computed for each area and a total score. The five areas are:
16
n Creativity—parental acceptance of the child’s creativity and willingness to encourage its
development;
n Frustration—parental frustration with the child and focus of the frustration;
n Control—parental feelings about the need to control the child’s behavior;
n Play—parental understanding of play and its influence on child development; and
n Teaching/Learning—parental views about child development and their ability to
provide a supportive home environment.
The second instrument is a behavior frequencies assessment (BFA), which was developed
specifically for the Family Literacy Pilot Program evaluation. The BFA is a self-report
instrument on which parents indicate how frequently they engage in ten different behaviors,
such as reading to their children and helping their children with homework.
Parents Enter Program with
Good Parenting Attitudes
Most of the parents in the Program entered with good parenting attitudes as measured by
the PAAT. Ninety-five percent of the first-year parents entered the Program with PAAT
scores of 125 or greater, the number which the PAAT developers recommend for determin-ing
if parents have “favorable” parenting attitudes. Eighty-six percent of the second-year
parents had pretest scores of 125 or greater on the PAAT. Since the overwhelming majority
of parents in the Program already have favorable parenting attitudes, it is unlikely that large
improvements can be made.
Parents Who Enter with
Unfavorable Parenting
Attitudes Make Large Gains
Parents most in need of the parenting support made significant gains on the PAAT. The
group of 34 parents who entered the Program with PAAT scores below 125 points gained an
average of 23 points. As a result, 24 of the 34 participants, or 71 percent, had scores above
125 at the posttest. These parents made much greater gains than the parents who already
had “favorable” scores. Their average improvement of 23 points compares to only 3 points
overall.
17
Most Parents Show Minimal Gains
in Parenting Attitudes As
Measured by the PAAT
Since most participants already had fairly good parenting attitudes, few gains were seen.
While parents in the 1995-96 school year showed improvements in parenting attitudes in
only one subscore and the total score of the PAAT, the second-year parents show very small
improvements in all of the subscores and in the total score.
First-year parents showed improvements in their control score from pre- to posttesting but
did not show significant improvements on the other four scores. However, the 159 parents
who were pre- to posttested during the first year showed small but statistically significant
improvement in their total score, which increased from an average of 136 to an average of
139 on a scale of 0 to 200.
Second-year parents made improvements in all five of the parenting areas measured by the
PAAT but the gains were small and would likely result in only minor changes in parents’
attitudes. For example, on the subareas with scores that have a 30-point range of 10 to 40, all
of the average gains are fewer than 2 points. (See Table 12, pages c-v through c-vi, in
Appendix C for the analysis of the gains.)
Behavior Frequency
Assessment Shows Little
Overall Change in Behaviors
Parents’ responses to the BFA show that they made few consistent improvements in the
behaviors assessed by this instrument. The only areas that show improvements in both areas
are going to school activities, and volunteering for school activities. While the parents were
more likely to read to their children at the end of the first year than they were at the
beginning, this positive outcome is not evident in the second year. Table 4 (see page 18)
summarizes the BFA changes.
Family Literacy and Even
Start Show Same Impacts
Analysis of the changes for Family Literacy Pilot Program participants and Even Start partici-pants
on both the PAAT and the behavior frequencies show that both programs have essen-tially
the same impacts on parenting skills. There are no statistically significant differences
between the two groups on any of the parenting assessment pre- to posttest results.
18
Table 4
Family Literacy Pilot Program
Pre- to Post-Program Changes in Parenting Behaviors
1995-96 and 1996-97 School Years
Behavior
Pre- to Post-Program Changes
1995-96 1996-97
Number of days per week parents:
Talk about children’s day Improved No difference
Read or look at book with children Improved No difference
Let children see them writing Improved No difference
Take children to the library No difference Improved
Play with children No difference No difference
Number of days per month parents:
Go to a school activity Improved Improved
Volunteer for a school activity Improved Improved
Help children with homework No difference Improved
Talk to children’s teacher No difference No difference
Talk to children about school No difference No difference
Source: Auditor General staff analysis of data provided by Family Literacy Program staff.
Parents Report Improved
Parenting Behaviors
A review of statements made by Family Literacy participants shows they believe the Pro-gram’s
PACT and parent time components helped them improve their parenting.
We learned how to work together, to share, and to be patient to each other. I learned to respect
my daughter’s decisions.
I learned how important is the attention from the parents to the children in their education.
We can learn how we can help our children, talk with the teachers, and how the education sys-tem
is. . .how we can educate our children.
While parents report that the program components have helped them in their parenting
skills, they also say their parenting behaviors have improved as a result of their changes in
attitudes about parenting:
19
The most important thing to me is that I don’t yell at my children anymore and I don’t hit
them because I love my children. Through this program I have been learning how to teach my
children, and I try to be a good parent.
Before I entered the program…I didn’t know how to listen to my children so I yelled at them
all the time. But now everything has changed. I am more patient, I understand that it is better
to talk and listen to them instead of yelling at them. When my children cry or ask for some-thing
I don’t get mad, I talk to them.
Conclusions
Overall, the Program’s parenting and PACT components have minimal impacts on
improving the parenting skills of the adults in the Program. However, for parents with the
greatest needs in this area, the Program has had significant impact on improving their skills.
The parents’ comments demonstrate that they believe the Program has had a positive
impact.
20
(This Page Intentionally Left Blank)
21
FINDING III
FAMILY LITERACY PRESCHOOL
PARTICIPANTS MAKING PROGRESS
The Family Literacy Program’s preschool component appears to keep at-risk children from
falling further behind their not-at-risk counterparts, and to a degree, it may be helping them
to catch up. Analysis of program children’s scores on a kindergarten readiness assessment
shows they had a developmental rate that averaged four months ahead of nonparticipating
children. Finally, a comparison of progress made by Family Literacy children with Even
Start children suggests the Family Literacy children may be doing slightly better.
Background
A major objective of the Family Literacy Pilot Program is to improve the school readiness of
the 3- and 4-year-old children in the Program by providing developmentally appropriate
preschool education.
In order to assess children’s progress toward school readiness, children were pre- and post-tested
on the Activity Evaluation System for Observing Preschoolers (AESOP). The as-sessment
was selected for evaluation primarily due to its developmentally appropriate
testing methodology, and because teachers enjoy giving it and are therefore more likely to
comply with the testing schedule and protocol. The AESOP provides a total score and
subscores in three areas: 1) problem solving; 2) gross motor skills; and 3) language develop-ment.
Age-equivalent scores are reported from a youngest range of 1 year, 6 months to 2
years, to an oldest category of 5 years, 6 months and older.
Data Analysis Reveals
Program Has Impact
on Children’s Development
AESOP scores for children after one year in the Program were compared to AESOP scores
for children who had not had the program experience. Using this method of analysis, it
appears that the program intervention that children receive increases their development by
1 to 7 months, or an average of 4 months above their expected developmental progress
without the Program. While children are still generally behind where they should be for
their age, the Program keeps them from falling further behind and reduces the differences in
performance between them and their non-at-risk peers.
22
Analysis was conducted on children in four age ranges.1 Children’s AESOP average (me-dian)
scores after having been in the Program were compared to scores for children of the
same age who had not been in the Program.
Children who had been in the Program had higher scores than children who had not been
in the Program. Gains in favor of the program children are found on all three subscores and
on the total scores. For the two younger groups of children, the gains in favor of the pro-gram
children were as much as seven months. For the two older groups of children, the
gains in favor of the program children range from one to three months. The children’s
performance, as measured in months, is presented in Table 5.
Table 5
Family Literacy Pilot Program
Activity Evaluation System for Observing Preschoolers
Median Score in Months by Age Group
1995-96 and 1996-97 School Years
Age Range in Months
Children without
Family Literacy
Participation
Children with
Family Literacy
Participation
Difference
Attributable to
Family Literacy
1995-96 School Year
33 to 46 41 48 7
47 to 51 44 51 7
52 to 57 47 51 4
58 to 66 51 52 1
1996-97 School Year
32 to 45 41 48 7
46 to 52 47 50 3
52 to 57 49 51 2
58 to 70 51 53 2
Source: Auditor General staff analysis of data provided by Family Literacy Program staff.
The average of 7 months of program time that children receive appears to increase their
development by 1 to 7 months above their predicted developmental progress without the
Program. Although program children are still generally developmentally behind for their
chronological age, the Program keeps them from falling further behind and may help them
to catch up to their non-at-risk peers.
1 Children were grouped into four quartile (groups of equal size whose values are mutually exclusive)
age ranges. The grouping allowed for adequate numbers for each analysis while also limiting the age
range being analyzed.
23
AESOP May Fail to
Measure Some Growth
AESOP may not fully measure growth for older children as well as it does for younger
children because of what is known as a “ceiling effect” of the instrument. There is a direct
relationship between the amount of improvement registered by AESOP and the children’s
average developmental age on the pretest. Because AESOP is a kindergarten readiness test,
it is best able to measure a child’s developmental growth when the child’s initial perform-ance
score is considerably below the norm for kindergarten. It is less sensitive to measuring
growth that may occur in children whose pretest developmental age is closer to a kinder-garten
norm.
Comparison Reveals Family Literacy
Children May Be Doing Better
Than Even Start Children
A comparison of the Family Literacy children’s gains to the gains made by Even Start chil-dren
reveals Family Literacy children made slightly better gains in language and on their
total AESOP scores than did the Even Start children. No differences in the gains are found
for the motor skills and problem-solving domains in the second year. Analysis for the 1995-
96 school year reveals that the Even Start group had slightly higher gains in problem solv-ing.
Conclusion
Children in the Family Literacy Program appear to make performance gains greater than
expected for an at-risk population of children. As a result of the Program they are reducing
the degree to which their pre-literacy skills are behind their non-at-risk peers’ and are more
likely to start school ready to learn.
24
(This Page Intentionally Left Blank)
25
FINDING IV
PROGRAM IMPLEMENTATION AND
MONITORING IMPROVES IN SECOND
FULL YEAR OF OPERATION
Since last year’s report, most contractors operating sites for the Family Literacy Program
have made progress in offering quality programs. Site evaluations showed that 20 of the 23
sites that were evaluated made progress in meeting statutory requirements and correcting
previous quality-related problems in such areas as integrating components, collaborating
with other providers, providing childcare, and maintaining enrollments at required levels.
The higher degree of quality and compliance reflects hard work on the part of the sites,
improved monitoring from ADE, and continued technical assistance from the model pro-grams.
Background
Last year’s evaluation report raised concerns regarding the sites’ failure to meet criteria
required by Arizona’s Family Literacy Pilot Program. The framework in which family
literacy programs must operate is clearly defined. The Children and Stability Act of 1994
established specific criteria for family literacy providers to meet. Contractors are required to
enroll a set number of families, enroll only families who meet specific eligibility criteria,
provide and integrate the four major components of a family literacy program, collaborate
with other agencies, and help families meet their childcare and transportation needs. Sites
are also contractually required to provide a minimum number of service hours per family.
During the first year of program implementation many sites did not meet statutory or
contractual criteria. Only 7 of 26 family literacy sites met all 8 requirements. Two of the sites
were closed in the second half of the 1995-96 school year due to their failure to provide
contracted services, and the other 17 sites did not fulfill between 1 and 4 requirements.
Last year’s evaluation also reported that ADE’s monitoring of most sites during the 1995-96
school year was inadequate. The few site visits ADE conducted were not formally struc-tured,
and ADE did not provide formal feedback to contractors regarding deficiencies. The
lack of adequate monitoring contributed to the compliance problems and poor program
operations found at many of the sites. Due to the relationship between monitoring and
program operations, it was recommended that ADE improve its monitoring procedures.
26
During the 1996-97 school year, Auditor General staff assessed program quality and adher-ence
to program criteria through visits to all 23 Family Literacy sites. Two visits were made
to 21 of the sites, and because of scheduling problems, 2 sites were visited only once. The
Site Observation Form and the Adult and Early Childhood Education Interview Rating
Form, described in Appendix A (see pages a-i through a-iv), were used to record data re-garding
program operations.
Sites Improve Compliance with
Statutory and Contractual Guidelines
In 1996-97, Family Literacy sites improved their compliance with program criteria specified
in laws and contracts. Programs are more conscientious about meeting the legislative and
recommended criteria and most programs made efforts to address compliance issues
discussed in last year’s evaluation report. However, a few sites still did not meet some of the
criteria.
Improvements were noted in much of the criteria—Improvements made by Family Literacy
sites are described as follows:
n Twenty-two sites are implementing programs based on standards specified in the
National Center for Family Literacy model and are making sure that the components
recommended in the model are integrated. While seven sites were noted to have problems
in this area last year, only one site still lacks integration.
n Twenty sites have ensured that they are providing at least 15 hours of service to families.
Two of the sites with problems in this area during the 1995-96 school year are now
offering at least 15 hours of service. Two other sites are close to the minimum, providing
over 14 hours per week.
n Twenty-two sites have made efforts to provide childcare for families with infants. Three
of 4 sites cited last year for lack of childcare have identified various resources and have
referred families for these services.
n All sites have strengthened relationships with collaborators by working with them to
improve and expand services. For example, Tucson High staff has fostered strong
support from the school principal to maintain the Program next year as one of two
remaining Tucson Unified School District (TUSD) sites. The Cochise College site
improved its early childhood component by collaborating with the local Head Start
provider.
n One of 2 programs cited for problems in eligibility screening has been more thorough
and did not have ineligible participants enrolled this year.
27
Table 6 (see pages 28 through 29) provides the instances of noncompliance for the 1996-97
school year in contrast to the instances found during the 1995-96 school year. Only Leupp,
Pinon, and Phoenix Indian Center remain out of compliance with 2 or more of the 8 criteria.
Some problems continue with eligibility and enrollment—Although most sites successfully
addressed participants’ eligibility, two still had families of questionable eligibility in the
Program. The Leupp site served families who, based on legislative requirements, were not
qualified for the Program since they had no age-eligible children. Without these families in
the Program, the site would be considered to have low enrollment. Pinon is another
program with families enrolled who are of questionable eligibility, since they do not have
age-eligible children; however, with the exclusion of these families the Program still meets
the enrollment criteria of a minimum of 10 families.
Enrollment continued to be a problem at some sites. Programs such as Leupp, Killip, and
Pinon are examples of sites that have had enrollment and attendance problems. Two
additional programs with questionable enrollment were closed for the 1997-98 school year
(Tucson Unified School District’s Pueblo Gardens and Lawrence sites).
ADE Improves Monitoring
Some of the improvements sites have made in compliance with Family Literacy criteria can
be attributed to ADE’s greater attention to site monitoring during the 1996-97 school year.
ADE staff visited all sites and during their visits paid specific attention to sites’ compliance
with program guidelines. ADE also followed up with sites after visits and communicated
frequently with sites that continued to have difficulties. ADE’s improved monitoring re-quired
sites to address compliance issues or face the possibility of having their contracts
terminated. Three sites began the 1997-98 school year on a month-to-month contractual
basis due to poor performance during the 1995-96 and 1996-97 school years.
In addition to improving its program monitoring, ADE implemented the Auditor General
recommendation to work more closely with the federally funded counterpart of Family
Literacy, the Even Start Program. Beginning in the 1996-97 school year, Even Start and Fam-ily
Literacy Program coordinators held joint meetings each month.
Model Sites Provide Valuable
Technical Assistance
The two Model Programs, Pima County Adult Education (PCAE) and Mesa Family Tree,
continued to provide valuable technical assistance to the sites. This technical assistance
further contributed to sites’ compliance with Family Literacy criteria.
Table 6
Family Literacy Pilot Program
Instances of Noncompliance by Site and Requirement
Years Ended June 30, 1996 and 1997
Site Enrollment Eligibility
Offers all 4
NCFL
Components1
Components
Integration Collaboration
Child
Care Transportation
Service
Hours Total
’96 ‘97 ’96 ‘97 ’96 ‘97 ’96 ‘97 ’96 ‘97 ’96 ‘97 ’96 ‘97 ’96 ‘97 ’96 ‘97
Crane ESD No. 13
Crane Morning Session X 1 0
Crane Afternoon Session X 1 0
Flagstaff USD No. 1
Killip X X 2 0
Leupp X s X s X X s 4 3
Isaac ESD No. 5
Isaac English for
Speakers of Other
Languages
X X s 2 1
Isaac GED X X s 2 1
Mesa USD No. 4
Eisenhower X X 2 0
Lincoln 0 0
Longfellow X 1 0
Pima County Adult
Education
Liberty 0 0
Nash 0 0
Ochoa 0 0
Pinon USD No. 4 X s X X s 3 2
Red Mesa USD No. 27 0 0
Somerton ESD No. 11 X X 2 0
Tempe ESD No. 3 X X X X s 4 1
Tucson USD No. 1
Lawrence X 1 0
Meyers-Ganoung X X 2 *
* Site closed. (Continued)
Table 6
Family Literacy Pilot Program
Instances of Noncompliance by Site and Requirement
Years Ended June 30, 1996 and 1997
(Concl’d)
Site Enrollment Eligibility
Offers all 4
NCFL
Components1
Components
Integration Collaboration
Child
Care Transportation
Service
Hours Total
’96 ‘97 ’96 ‘97 ’96 ‘97 ’96 ‘97 ’96 ‘97 ’96 ‘97 ’96 ‘97 ’96 ‘97 ’96 ‘97
Tucson USD No. 1 (cont’d)
Pueblo Gardens 0 0
Tucson High School X X 2 0
Wakefield 0 0
Other
Cochise Community College X X X X 4 0
Literacy Volunteers of
Maricopa County X X X X 4 0
Phoenix Indian Center X s s s 1 3
Total 12 3 2 1 5 1 7 2 1 0 4 1 2 0 5 3 38 11
1. Adult Education, Early Childhood Education, Parent Time, and Parent and Child together.
Note: Chino Valley USD No. 51 and the Council for Family Concern in Willcox were closed and are not included in the table.
X = Noncompliance in 1996.
s= Noncompliance in 1997.
Source: Auditor General staff analysis of data provided by the Arizona Department of Education and Family Literacy Program contractors.
30
All the Family Literacy sites are serviced by one of the two training resources. The Pima
County Adult Education program trainers provide family literacy support services and
training to sites in southern Arizona, and the Mesa program assists sites in Phoenix and
northern Arizona. The trainers also work with ADE to provide statewide training for new
and returning program staff.
Program coordinators, administrators, and staff were interviewed regarding their opinions
about the technical assistance. Overall, the staff’s responses to technical assistance services
are positive. Most found the services helpful and indicated the model programs helped
them to implement the Program in accordance with the statutory guidelines.
The model programs have an advantage over any ADE staff in providing technical assis-tance.
While ADE staff can provide guidance, unlike the model programs, they do not have
direct, current experience operating family literacy programs. If the Family Literacy Pro-gram
is expanded to new sites, the model programs could provide valuable technical assis-tance
and training to the staff at new sites. They also can continue to provide training to
established sites as staff turnover occurs. The ADE and model programs have positively
responded to the recommendation in last year’s report to collaborate in the development of
a monitoring plan and site visits. As a result, the model programs also provide some assur-ance
that sites are operating in compliance with the Program’s criteria. They help programs
come into compliance in such areas as enrollment criteria, collaboration, and the integration
of program components.
Quality Improves at
Family Literacy Sites
As a result of sites’ improved compliance with criteria, ADE’s improved monitoring, and
the ongoing support provided by the model programs, quality has improved at most family
literacy sites throughout Arizona. Auditor General staff rated the 23 sites visited on the basis
of 82 quality standards, such as whether the adults’ lessons include regular large- and small-group
activities, paired learning, and peer tutoring, and whether children can choose their
own activities during specific times of the day. Twenty of the sites have improved the
quality of services offered to program participants. Sites with prior family literacy
experience continue to do well and some new sites have also made gains in the quality of
services they offer. However, some other sites are not doing well despite technical assistance
and ADE monitoring.
Quality closely related to compliance—As sites increased their compliance with Family
Literacy Program criteria, the quality of services offered to participants improved. For
example, as sites offer daycare for younger children of program families, lack of daycare as a
deterrent to program participation is eliminated. Sites’ increased emphasis on collaboration,
such as the Cochise College site’s collaboration with the local Head Start provider, resulted
in improvements.
31
Sites that have operated for several years offer quality services—Programs that have
operated for several years tend to have excellent teachers and strong teams to provide
quality literacy services. Even when there is staff turnover at these sites, new staff is skilled
at continuing where the last teacher left off. These programs have established reputations in
their communities and utilize their experiences and resources well to recruit and retain
families. Most of these sites tend to have waiting lists as opposed to newer sites that struggle
to recruit. The experiences and reputations these sites have are an advantage over those who
are new at providing family literacy services.
New sites make improvements—There are also some new sites doing well this year, such as
Literacy Volunteers of Maricopa County, Tempe, and Cochise. At these sites, improved
collaboration and the addition of new staff have contributed immensely to the programs’
improved quality. Last year these three sites had many quality problems and low enroll-ment.
Inexperienced staff and poor program implementation were contributing factors, but
these problems have now largely been resolved.
Some programs continue to struggle—Three of the 23 sites evaluated, Pinon, Leupp, and
Phoenix Indian Center, are not doing well. These sites have made staff changes but continue
to struggle with quality and enrollment. ADE has made efforts to improve the programs,
and all three sites have developed plans to implement changes for improvement. These are
the programs that have been refunded for the 1997-98 school year on a month-to-month
contract instead of the quarterly payment plan the other programs use.
Recommendations
If the Legislature continues or expands the Family Literacy Program, the following should
be included in the legislation:
1. ADE should be required to continue program monitoring and technical assistance efforts
to ensure program quality remains.
2. Current statutory provisions providing for two existing programs to serve as models
and training resources for other sites should be retained.
32
(This Page Intentionally Left Blank)
33
STATUTORY
EVALUATION COMPONENTS
Pursuant to Laws 1994, Ninth S.S., Chapter 1, '9, the Office of the Auditor General is re-quired
to include the following information in the annual program evaluation.
C.1. Information on the number and characteristics of the program participants.
The participant information that follows is presented in composite from all 23 sites
for the 1996-97 school year. The demographics are very similar to the demographics
that were reported in Report No. 96-20 for the 1995-96 school year.
n Program Families—Typically, the family members directly served by the Family
Literacy Pilot Program include a mother and her three- or four-year-old child.
Occasionally a mother may have more than one child in the preschool component
(a three- and a four-year old, or twins). In addition, there are some cases in which
a grandparent, aunt, or father is served by the adult education component. There
are also several cases of more than one adult from the family being enrolled in the
adult education component.
n Program Parents—Most family literacy adult participants are mothers, though
there are a few who are fathers, a grandparent, or an extended family member.
Eight percent of the adults in the program are men. Participants’ ages range from
18 to 59. The median age is 29, and 95 percent are 40 or younger.
n Program Children—One-third of the children served are three-year-olds and
two-thirds of the children are four-year-olds. Eighty percent of the families in the
Program have three or fewer children.
n Number of Enrolled Participants—Throughout the 1996-97 school year, 23 sites
enrolled at least 482 families. By statute, family literacy sites target between 10
and 15 families per site. In an attempt to maintain the minimum enrollment of 10
families, sites enrolled participants continuously throughout the year as enroll-ment
fluctuated due to attrition. The 482 participants did not necessarily com-plete
a full year in the Program. The data reflects participants’ status at enroll-ment.
Several of the participants returned to the Program from the 1995-96 school
year.
n Family Ethnicity—The majority of families enrolled are Hispanic (78 percent).
Other ethnicities represented are Native American (13 percent); White, non-
34
Hispanics (4 percent); African-American (1 percent); and Asian/Pacific Islanders (1
percent). “Others” or “unknown” make up the balance. English language literacy is
low, with only 18 percent of the families reporting English as the primary language
spoken at home and an additional 10 percent reporting English as one of the lan-guages
spoken at home. Two-thirds report they speak only Spanish at home. Na-vajo
and other languages are spoken at home in a small percentage of families.
n Family Status—Participants come from poor and undereducated backgrounds.
Fifty percent of the families have incomes of less than $10,000, and over half of the
adult participants have a ninth-grade education or less.
While almost 80 percent of the participants’ spouses (mostly male) work and provide
the primary source of income for the family, only 13 percent of adult participants are
employed. Twenty-eight percent of the families rely, at least partly, on some govern-ment
assistance.
Two-thirds of the participants in the family describe their living situation as a couple
with children. An additional 17 percent live in extended families, and only 16 percent
are single parents with child(ren).
C.2. Information on contractors and program service providers.
n Fifteen Original Pilot Program Contractors—The State Board of Education ini-tially
approved family literacy project monies for 15 contractors serving 26 sites in
Arizona. Contractors included 10 school districts, 3 community-based organiza-tions,
1 college, and 1 adult education provider. These contractors provided serv-ices
during the 1995-96 school year. For 1996-97, 13 contractors were serving 23
sites. ADE did not renew contracts for Chino Valley Unified School District’s and
Willcox’ Council for Family Concern’s sites for 1996-1997, and one site was closed
at the request of the contractor, the Tucson Unified School District (TUSD). For the
1997-98 school year, the same 13 contractors are providing services at 21 sites. The
TUSD also requested the closure of 2 sites at the end of the 1996-97 school year.
Contract award amounts are presented in Table 1 (see page 5 in the Introduction
and Background).
n Two Model and Training Resource Contractors—Mesa Unified School Dis-trict’s
Family Tree Project and Pima County Adult Education served as model
and training programs. Both models also served as contractors for a total of 6 sites
during the 1995-96 and 1996-97 school years. Mesa Unified School District serves
12 sites in Arizona’s northern region (including Maricopa County), and Pima
County Adult Education serves the remaining 11 southern Arizona sites.
35
n Program service providers given extensive opportunities for family literacy
training—Prior to the beginning of the 1995-96, 1996-97, and 1997-98 school
years, Arizona’s Family Literacy Pilot Program staff received training in imple-menting
a family literacy program. ADE’s Adult Education Division collaborated
with the two model programs, the State’s Even Start coordinator, and the Arizona
Adult Literacy and Technology Resource Center, Inc., to provide a weeklong
summer institute. The comprehensive statewide training included Family Liter-acy
Pilot Program and Even Start staff, program coordinators, and administra-tors.
The summer 1995 training was supported with state Family Literacy monies
and the summer 1996 and summer 1997 training with federal adult education
and Even Start monies.
The extensive trainings were supplemented by training sessions conducted in June,
August, and December 1995, and January 1996, and by on-site training provided by
the model programs’ staff.
C.3. Information on program revenues and expenditures.
Appropriations for the Family Literacy Pilot Program total $975,000 in school year
1994-95 and $1,000,600 in school year 1995-96. School years 1996-97 and 1997-98, re-spectively,
have been appropriated $1,000,000. Family Literacy Pilot Program monies
for 1994-95 were nonreverting, allowing ADE to carry over unspent monies.
ADE reduced 1995-96 contracts’ budgets by 12 percent to correct budgeting errors
made in school year 1994-95. Sites were forced to revise their budgets mid-year to
operate on the reduced monies. Program budgets for 1996-97 were cut an additional
24.5 percent from the 1995-96 levels.
C.4. Information on the number and characteristics of enrollment and disenroll-ment.
The 23 sites served 482 families over the course of the 1996-97 school year. Of these
families, 213 families have pre- and posttest data indicating that they successfully
completed their goals for the Program. The sites have retained 54 percent1 of the
families long enough for pre- to posttesting; however, there is great variability among
these sites as far as their success in retaining participants. Of the 3 sites that have been
problematic, 1 has the lowest retention rate, only 18 percent, and the other 2 did not
submit the end-of-year data required to compute the retention rate. Sites that have
been in existence for a long time tend to have better retention rates. For example, 2 of
1 Percentage is based on sites with complete posttest and exit data. Since some participants entered the
Program too late for both pre- and posttesting, the reported rate is modestly deflated.
36
Pima County Adult Education’s sites have very high retention rates, 89 and 80 per-cent.
One of the Mesa sites has a high rate of 67 percent, and Crane and Isaac, who
have operated programs since the early 1990s, have rates of 69 and 75 percent, re-spectively.
One new site, Red Mesa, also has a commendable rate of over 75 percent.
Enrollment information for each site in operation during the 1996-97 school year is
summarized in Table 7 (see page 37).
Families who exited the Program prior to the end of the school year did so for a vari-ety
of reasons. Almost half of the early exits were for positive reasons. For example,
of the 289 families who exited prior to completion of the Program, almost 30 percent
did so because they had met the adult participants’ goals. Another 16 percent left due
to employment. However, 21 percent left because of a lack of interest or poor atten-dance.
Some type of family crisis, such as illness or other conflict, caused another 20
percent to leave the Program before the end of the school year. The balance of the
terminations was due to a variety of other reasons.
The overall retention rate for 1996-97 is improved over the 1995-96 rate of 40 percent.
However, some sites did not make improvements in retention. Somerton and Tempe
saw decreases in their retention rate, as did the Ochoa model site. Overall, however,
the Program’s higher retention rate indicates improvements in program operations.
Retention rates for the Family Literacy Sites are similar to family literacy programs
across the country. National Center for Family Literacy (NCFL) reports that pro-grams
across the country have a 50 percent rate of attrition, and national studies of
program effects are reported on an average of only 10 percent of the families with
matched pre- and posttest scores.
C.5. Information on the average cost for each family in the Program.
The Family Literacy model is based on the premise that a program incorporating
adult, child, and parent education is more effective than independent adult or early
childhood components. The complexity and variety of services provided by a family
literacy program requires multiple resources. In addition, budgets required to oper-ate
a family literacy program can vary depending on the number of days per week
the program operated, the program size, and the cost of conducting the program in a
community. NCFL estimates a full-time program requires a budget of $50,000 to
$90,000 per site per year.
Cost per family—Family literacy programs spent a median of $4,024 of state monies
per family for family literacy services during the 1995-96 school year and $3,038
during the 1996-97 school year.
37
Table 7
Family Literacy Pilot Program
Number and Percentage of Families
Who Completed Programs per Site
1995-96 and 1996-97 School Years
(Unaudited)
1995-96 1996-97
Site
Number
Completed 1
Number
Served
Percentage
Completed
Number
Completed
Number
Served
Percentage
Completed
Crane ESD 13 (2 classes) 19 25 76% 18 26 69%
Flagstaff USD No. 1
Killip 5 12 42 12 22 55
Leupp 2 1 25 4 -- 23 NA
Isaac ESD No. 5 (2 classes) 28 49 57 24 32 75
Mesa USD No. 4
Eisenhower 4 25 16 11 29 38
Lincoln 6 23 26 11 28 39
Longfellow 12 18 67 16 24 67
Pima County Adult
Education
Liberty 10 22 45 16 18 89
Nash 12 26 46 12 15 80
Ochoa 14 19 73 6 16 38
Tucson USD No. 1
Lawrence 13 30 43 5 11 45
Pueblo Gardens 11 32 34 6 20 30
Tucson High School 4 19 21 6 18 33
Wakefield 9 34 26 13 27 48
Contractors with single
classroom sites
Cochise Community
College 11 33 33 11 19 58
Literacy Volunteers of
Maricopa County 13 35 37 11 23 48
Phoenix Indian Center 2 6 27 22 -- 13 NA
Pinon USD No. 4 7 28 25 7 39 18
Red Mesa USD No. 27 20 30 67 22 29 76
Somerton ESD No. 11 11 15 73 11 28 39
Tempe ESD No. 3 12 28 43 6 22 27
Total 228 555 41 224 482 503
1 Based on additional data, totals were adjusted from the number reported in Auditor General Report
No. 96-20.
2 Completion data was not submitted for the 1996-97 school year.
3 Excludes sites with no completion data.
Source: Auditor General staff analysis of data provided by Family Literacy Program staff.
38
Auditor General staff calculated the cost per family using state monies allocated to
contractors for school years 1995-96 and 1996-97 and the estimated number who
completed services each year at each site.
Costs vary substantially across programs—Cost per family using state monies allo-cated
to contractors for school years 1995-96 and 1996-97 varied considerably across
sites. Variation is attributable to the different amount of monies allocated to each site
and the number of families served. Costs per family for each site are presented in Ta-ble
8 (see page 39).
Family literacy programs cost an average of $6,830 in state monies per family for
family literacy services during the 1995-96 school year and $4,208 during the 1996-97
school year. However, there are 3 programs with typically high costs in the 1995-96
school year, and 1 such program in the 1996-97 school year. Excluding these atypical
sites from the calculation reduces the average cost to $4,567 for the 1995-96 school
year and $3,765 for the 1996-97 school year.
For some sites, the estimated costs may reflect the total costs for the Program. How-ever,
many sites collaborate with other providers. For example, Literacy Volunteers
of Maricopa County collaborates with Head Start for early childhood education
services. The costs for this site reflect only about half of the actual program costs.
However, Somerton relies more exclusively on the state monies for their program
and their costs reflect most of the actual program costs.
High program costs are generally associated with programs that have been identified
as having implementation difficulties. For example, Leupp and Phoenix Indian Cen-ter
had high costs per family for 1996 and did not provide data for 1997. These sites,
along with Pinon, which has high costs for both years, are receiving extra scrutiny by
ADE to ensure they meet their contractual obligations. Two other sites that had high
1997 costs, Lawrence and Pueblo Gardens, have been discontinued for the 1997-98
school year at the request of the contractor, Tucson Unified School District No. 1.
Comparable programs’ cost comparisons—Cost comparisons with other similar pro-grams,
such as the federally funded Arizona Even Start, show those programs to be
providing family literacy services at a cost similar to Arizona Family Literacy Pilot
Program sites during the second full year of implementation.
Arizona Family Literacy Pilot Programs and Even Start both deliver family literacy
services to parents and their children. The differences between the two programs are
their sources of financial support, and their age eligibility requirements for children.
Even Start is a federally supported program that provides services to families with
children ages 0 to 7, while the Arizona Family Literacy Pilot Program serves families
with three- and four-year-old children. Additionally, Even Start has no class size
limitation or citizenship eligibility requirement.
39
Table 8
Family Literacy Pilot Program
State Cost per Family per Site 1
Years Ended June 30, 1996 and 1997
(Unaudited)
Site 1996 1997
Crane ESD No. 13
Crane morning session $ 3,961 $ 3,157
Crane afternoon session 3,961 3,157
Flagstaff USD No. 1
Killip 7,489 2,356
Leupp 37,444 NA
Isaac ESD No. 5
Isaac English for Speakers of Other Languages 3,143 2,768
Isaac GED 3,143 2,768
Mesa USD No. 4
Eisenhower 8,962 2,529
Lincoln 6,141 2,529
Longfellow 3,071 1,739
Pima County Adult Education
Liberty 4,400 2,076
Nash 3,666 2,768
Ochoa 3,143 5,536
Tucson USD No. 1
Lawrence 3,385 8,305
Pueblo Gardens 4,000 6,921
Tucson High School 11,000 6,921
Wakefield 4,889 3,194
Contractors with single classroom sites
Cochise Community College 4,024 3,038
Literacy Volunteers of Maricopa County 3,082 2,750
Phoenix Indian Center 7,206 NA
Pinon USD No. 4 17,317 13,075
Red Mesa USD No. 27 4,397 3,018
Somerton USD No. 11 5,600 4,228
Tempe USD No. 3 3,667 5,533
Average $ 6,830 $ 4,208
1 The actual cost per family was calculated using the number of families who completed the Program. The
actual cost per family reported in Auditor General Report No. 96-20 was calculated using the average
number of families served per month.
Source: Auditor General staff analysis of data provided by Family Literacy Program site staff.
40
Arizona’s federally supported Even Start programs closely resemble the Family Lit-eracy
Pilot Program sites; in fact, there is collaboration between Family Literacy Pilot
Program and Even Start sites. Programs collaborate to reduce service costs and in-crease
available resources, and both programs adhere to National Center for Family
Literacy models. The costs for the two programs are very similar.
Other comparable programs in the country, such as the federally supported Head
Start Family Service Centers, privately supported Kenan Family Literacy Project, and
PACE, provide similar services to families at costs comparable to Arizona’s Family
Literacy Pilot Program. As seen in Table 9 (see page 41), cost comparisons show that
these programs provide services at costs very similar to the average for Arizona’s
1996-97 pilot sites. Table 9 also shows the costs per family for these and the family lit-eracy
programs.
C.6. Information concerning progress of program participants in achieving goals
and objectives.
Finding I (see pages 9 through 14), describes the progress adult participants have
made in improving their literacy skills, attaining GEDs, and obtaining employment.
We report that adults have made significant improvements in their literacy skills, a
small number have successfully completed their GED, and many more are continu-ing
to work toward their GED. Additionally, more participants are employed.
In Finding II (see pages 15 through 19), modest improvements in adults’ parenting
skills are reported.
Finding III (see pages 21 through 23), discusses the developmental gains the partici-pating
children are making as a result of the Program. Children who participate in
the Program show a developmental rate that is an average of 4 months ahead of
nonparticipating children as a result of their participation in the Program.
In addition to the progress reported in Findings I through III, participants have ac-complished
other outcomes. For example, 88 participants received library cards, 40
participants earned their First Aid card, and 23 participants received CPR certifica-tion.
One participant became a U.S. citizen while in the Program, and 23 worked on
acquiring their citizenship. Four other participants became newly registered voters
while in the Program.
41
Table 9
Family Literacy Pilot Program
Cost of Comparable Programs
1995-96 and 1996-97 School Years
Program
Average Annual Cost
per Family
1996 Family Literacy Pilot Program $6,830
1997 Family Literacy Pilot Program 4,266
Arizona Even Start (includes local effort) 4,975
Head Start Family Service Centers 3,507
Kenan Family Literacy Project 4,000 to 6,060
Source: Auditor General staff analysis of data provided by the Arizona Department of Education,
Family Literacy Program contractors, and information published on other programs.
C.7. Recommendations regarding program administration.
In our first annual evaluation (Auditor General Report No. 95-20), we recommended
that:
n The Legislature should consider increasing the number of families to be served at
each site from 10 to 15 families to 15 to 20 families.
This recommendation has not yet been implemented. Implementation of this rec-ommendation
could make the Program more cost-effective, while maintaining an
appropriate educational environment. Programs can serve up to 20 families at almost
the same cost it takes to serve between 10 and 15 families. The range increase would
thus allow programs to serve more families at a decreased cost per family. The in-crease
would also be in keeping with guidelines for early childhood programs for 3-
and 4-year-olds. We again make this recommendation in order to reduce the Pro-gram’s
costs per family.
In Finding IV (see pages 25 through 31), we recommend that the Legislature
consider the following if the Program is continued or expanded:
n ADE should continue its program monitoring and technical assistance efforts to
ensure program quality remains.
42
n Model program activities should continue in order to ensure that new and con-tinuing
sites have the assistance they need to succeed.
The Family Literacy Pilot Program initially had a difficult start-up period but has
improved its program administration and delivery. In addition, the Program’s long-term
impact has not been measured. In order to ensure the Program continues to
make an impact, and to assess long-term outcomes, ongoing evaluation of the Pro-gram
is warranted The ADE has continued all of the data collection that was initiated
by the Office of the Auditor General. In addition, the Auditor General’s staff have
met with ADE staff regarding a system for electronically capturing the data. With the
data collection already in place, it is recommended that:
n The ADE report annually, through December 31, 2002, to the Legislature on the
Program’s impacts on improving adult literacy, parenting skills, and the devel-opmental
progress of the children in the Program.
n The ADE report annually, through December 31, 2002, to the Legislature on the
educational and employment status of the adults and the educational progress of
the children who have been out of the Program for 6 months to 3 years.
C.8. Recommendations regarding informational materials distributed through
the programs.
The model programs have provided pilot programs with training and material
useful in the program implementation and no recommendations specific to the
informational materials are made.
Model programs disseminated family literacy manuals developed by the Na-tional
Center for Family Literacy (NCFL) to all pilot program staff. The manuals
familiarized staff with family literacy components and provided a comprehen-sive
guide for pilot program service delivery. NCFL manuals include extensive
information on topics such as recruitment and retention, adult education, early
childhood education, parenting discussion groups, parent and child interaction
time, and assessment and program evaluation.
Staff also received information and materials specifically relating to each of the
four program components. For example, adult education materials included
GED study books, pre-GED books, and reading books at various proficiency lev-els.
Early childhood education materials included children’s books and devel-
43
opmentally appropriate toys. In addition, staff received materials through ven-dor
fairs involving numerous adult and early childhood vendors, nutrition
classes offered by the University of Arizona, early childhood training from a na-tionally
recognized organization, and educational computer software.
C.9. Recommendations pertaining to program expansion.
The Family Literacy Pilot Program has positive impacts on the families who choose
to enroll into and stay in the Program. The Program appears to be successful in im-proving
the literacy rates of adult participants, and has some positive effects on their
parenting behaviors. In addition, preschoolers in the Program are making gains in
school readiness. Further, these gains were made despite a number of problems in
the manner in which the Program was implemented.
The evaluation has focused on benefits to the family members who have directly
participated in the Program. It should be remembered that the model is designed to
benefit not just the participating members, but all family members. Through im-proved
adult (parent) literacy skills, all the children in the home can be expected to
make academic gains. And, as families improve their income and self-sufficiency,
there will be additional benefits to the family. While improvements for other family
members have not been directly measured, it is assumed that such benefits have and
will occur. The benefits to the entire family, and not just the direct benefits measured
by the evaluation, are taken into account in making recommendations about pro-gram
continuation and expansion.
In Finding I (see pages 9 through 14), we report that the Program is successful in im-proving
the literacy rates of the adult participants. Although it is too early to measure
long-term effects, the positive adult education outcomes offer the possibility that the
Program may have the intended impact of breaking the intergenerational cycle of
poverty and illiteracy. As the parents’ educational and literacy skills improve, it is
expected that their children will do better in school. Not only should the preschool
children in the Family Literacy Program do better, but their older and younger sib-lings
should also do better.
In Finding II (see pages 15 through 19), we report that most parents enter the
Program with positive parenting attitudes and that the parenting and PACT
components of the Program only minimally impact their parenting skills. However,
those parents still demonstrate that they better understand their role as their
children’s first teacher and some are engaging in more positive forms of discipline, as
well as developing a better understanding of the education system and how they can
help their children. A small group of parents do enter the Program with unfavorable
parenting attitudes. Approximately 70 percent of those parents show significant
improvement while in the Program.
44
In Finding III (see pages 21 through 23), we report that the preschool children in the
Family Literacy Program appear to make performance gains greater than predicted
for an at-risk population of children. As a result of the Program, they are “catching
up” to their non-at-risk peers and are more likely to start school ready to learn. The
amount of “catching up” they are doing might be greater than what is actually
measured, since they would be expected to fall further behind if they were not in the
Program.
In Finding IV (see pages 25 through 31), we report that sites have made
improvements in complying with program criteria and offered higher quality
services during the 1996-97 school year than in previous years. Serious problems
with program implementation were found during the 1995-96 school year. Some
of the improvements can be attributed to the better monitoring and technical
assistance provided by the ADE and to ongoing services by the model programs
that have paid more attention to compliance with program criteria.
In the statutory section the Program’s costs are contrasted to other family literacy
programs and are found to be comparable.
Based on these positive outcomes from the Program that occurred despite problems
with implementation, it is recommended that the Program be continued and expan-sion
to additional sites considered. Positive results are being seen even though there
were serious problems during the first two years of program implementation. As
noted in our second evaluation of the Program last year (Report No. 96-20), only 7 of
26 programs complied with all program criteria. The second-year evaluation warned
that the problems with implementation threatened the Program’s potential for suc-cess.
The fact that positive impacts were found despite the implementation problems
suggests that greater impacts may be found in the future if the implementation
problems continue to be addressed.
However, if the Program is continued and possibly expanded, because Arizona’s
family literacy model cannot meet the needs of all adults and children in need of lit-eracy
services, any additional monies should be directed to the areas with the great-est
opportunity to make an impact. Monies should be distributed through a request
for proposal process requiring potential contractors to demonstrate high rates of eco-nomic
and educational disadvantage in their service area and a workable plan to at-tract
and retain the target population.
It is also recommended that the model program sites be continued in order to assist
with training and technical assistance needs of ongoing programs as they experience
staff turnover, and new sites as they begin to provide services. Model programs
should provide technical assistance, training, and basic monitoring in cooperation
with ADE staff.
45
Pursuant to Laws 1994, Ninth S.S., Chapter 1, '9, the Office of the Auditor General is re-quired
to include the following information in the final program evaluation.
E.1. Statistical information measuring the effectiveness of the programs in accom-plishing
the goals and objectives established in this act.
In Finding I (see pages 9 through 14), statistical information specific to the Program’s
effectiveness in improving the literacy skills of adult participants is provided.
In Finding II (see pages 15 through 19), statistical information specific to the Pro-gram’s
effectiveness in improving the parenting skills of adult participants is pro-vided.
In Finding III (see pages 21 through 23), statistical information specific to the Pro-gram’s
effectiveness in improving the school readiness of preschool participants is
provided.
In Appendix C (see pages c-i through c-vii), additional statistical information sup-plementing
Findings I, II, and III is presented.
E.2. The attitudes and concerns of program participants.
In Findings I and II and the Introduction, some of the participants’ attitudes regard-ing
the Program are presented through quotes from their written work. These quotes
generally reflect the participants’ attitudes concerning the assistance they have re-ceived
from the Program in regard to improving literacy, parenting skills, and help-ing
their children to learn. The participants acknowledge gains they have made in
these areas.
F.1. Evaluate the educational process for parents on developmental assessments
so that early identification of any learning disabilities, physical handicaps, or
behavioral health needs are determined.
Family Literacy, unlike Health Start and Healthy Families, the companion pro-grams
created with the Family Stability Act of 1994, does not focus on educating
parents as to the early identification of learning disabilities, physical handicaps,
or behavioral health needs. The Program does provide developmental assess-ments
of three- and four-year-olds and information from these assessments may
be used to refer children for additional early intervention services.
46
F.2. Measure the effects on program participants of promoting family unity and
strengthening family relations.
Qualitative data collected during participants’ interviews and through review of
written materials suggest the Program has improved family relations. For exam-ple,
one participant writes of the increased involvement by her husband in par-enting
while another states that her husband has become a better father. Partici-pants
routinely report improved relations with their children.
F.3. Review the impact on program participants of the counseling and coping
support services received.
Family Literacy does not provide direct counseling services. Through parent time,
participants receive coping support from their instructors, from outside presenters,
and from other participants. Qualitative data collected during participants’ inter-views
and through review of written material suggests the Program has provided
valuable coping support services for some participants. Participants report their fel-low
classmates helped them develop the skills necessary to effectively address and
deal with family and economic difficulties.
F.4. Evaluate the method for selecting eligible participants.
Family Literacy participants are required to 1) have a three- or four-year-old
child; 2) lack sufficient mastery of basic educational or basic English language
skills needed to function in society, or lack a high school diploma or its equiva-lent;
and be 3) United States citizens or legal residents, or otherwise be lawfully
present in this country.
Sites use a variety of methods to recruit program participants. Methods include ac-tivities
such as door-to-door recruitment, school districts referring families of pre-schoolers
if the parent(s) meet the criteria, and recruitment through local stores and
community centers and churches. Once established in communities, sites rely on
participants referring friends and relatives to the Program and general word of
mouth. Problems with low enrollments reported in Finding IV (see pages 25 through
31), and in the Statutory Annual Evaluation components suggest that sites need to
improve their recruitment and retention efforts and enroll participants who are a
good match for the Program.
Methods of determining eligibility vary across sites. While some sites require formal
documentation to ensure participants are United States citizens or legal residents,
other sites accept participants’ written or verbal assurance that they are citizens. As
reported in last year’s evaluation, some sites have enrolled adult participants who do
47
not meet the criteria, and in both last year and this year’s reports it is noted that some
sites have enrolled families without age-eligible children.
The ADE should provide guidelines for eligibility determination by local sites. The
guidelines should specify the types of documentation necessary for proof of citizen-ship
or legal residence, and clear guidelines for determining if a participant meets the
educational criteria. Additionally, the ADE should have guidelines defining allow-able
relationships between the adult and child in the Program and the maximum
number of adults per child who can be served through the Program.
F.5. Evaluate the overall effectiveness of the program based on performance-based
outcome measurements, including a reduced dependency on welfare,
increased employment, and increased self-sufficiency.
As reported in Finding I, many Family Literacy participants have spouses who
are employed, and therefore the participants themselves are focusing on raising
families and are not currently seeking employment. However, two-thirds of those
participants seeking employment have entered the workforce.
Due to problems with data collection, reliable data on participants’ welfare de-pendence
was not available.
Agency Response
1
December 9, 1997
Mr. Douglas R. Norton
Auditor General
State of Arizona
2910 North 44th Street
Suite 410
Phoenix, AZ 85018
Dear Mr. Norton:
I am responding to your agencies final evaluation of the Family Literacy Pilot Program.
A meeting to review the report draft was held, in our office, with members of my staff on
December 2, 1997. The discussion was very beneficial.
The report contains six recommendations for program improvement and our agency
addresses the recommendations as follows:
· Adult participants and their children, who were enrolled in the
family literacy program, will be required to complete follow-up inquiries
through longitudinal studies. This output measure study will be conducted
four times during a twenty-four month period after the family has finished
or withdrawn from the program.
The first effort to determine program impact is presently underway through
a research study conducted in Arizona by the National Center for Family
Literacy.
2
Results of the primary and secondary group sites will be available to your
office by January 5, 1998.
(Reference page 14)
· The Department will continue program monitoring through a
process of on-site visitations, program evaluation and technical assistance
workshops. In addition, institutes for administrators and instructional staff
will be conducted by the Department and the Arizona Academy for
Literacy Resources and Professional Development. Each program will
have a site visitation for technical assistance purposes no less than twice
each contract year.
(Reference: page 31)
· The Department will recommend to the legislature the continuation
of model training resources as presently required in Section E of the Family
Literacy Pilot Program.
(Reference: page 31)
· The Department will recommend to the legislature that Section I-5
of the Family Literacy Pilot Program be changed to read; ten but no more
than twenty parents…
(Reference: page 41)
· The Department will issue a report annually to the State Board of
Education and the legislature concerning program impact on parents and
children.
(Reference: page 42)
· With the passage of family literacy legislation in the 1998 session of
the legislature, the Department will expand the program to include
additional sites. Applications for a Family Literacy grant award will follow
the requirements of the Request for Proposal (RFP) process.
(Reference: page 44)
3
Again, our agency extends our appreciation to your staff for the outstanding assistance
provided in the evaluation of the Family Literacy Pilot Program.
Sincerely,
Lisa Graham Keegan
Superintendent of Public Instruction
Cc: Kathryn Kilroy
Jennifer Mabry
Gary A. Eyre
Lisa Rundle
1
1 Kk9
(This Page Intentionally Left Blank)
Appendix A
(This Page Intentionally Left Blank)
a-i
APPENDIX A
FAMILY LITERACY ASSESSMENTS
Background
The assessments selected and developed for the evaluation of the Family Literacy Pilot
Program were chosen to provide valid and reliable evaluations of the Program, and because
they are easy to administer and useful to the service providers. The assessments were se-lected
by the Office of the Auditor General staff in cooperation with ADE staff including the
state Even Start Coordinator, Family Literacy Coordinator, and the Director of Adult Educa-tion.
In addition, staff from the Pima County Adult Education Model Program and the Mesa
Family Tree Model Program were involved in the decision making. Finally, staff from the
Arizona Adult Literacy and Technical Resource Center provided assistance, along with
preschool educators from the Isaac Elementary School District.
Assessments of Adult Literacy
Adult Language Assessment Scales (LAS): CTB Macmillan/McGraw-Hill, Monterey, California.
1993. The Adult LAS reading and writing test is made up of two subcomponents consisting
of a total of five sections. The first three assess specific reading skills and are administered in
a multiple-choice format. The writing section is made up of two sections and is scored
holistically. The reading section includes 1) vocabulary, which measures the ability to match
pictures to English words; 2) fluency, which measures overall language fluency and ability
to infer a word or phrase based on knowledge of American English language usage and
semantics; and 3) reading for information, which measures the ability to gather information
from and answer questions based on printed materials. The writing section includes 1)
sentences in action, which measures the ability to write original sentences in English; and 2)
adventures in writing, which measures the ability to write an expository essay in English.
The LAS is designed to assess the probability of students’ success in the American main-stream.
Separate reading and writing scores are provided along with a combined score.
The LAS can be individually or group administered. The LAS has two forms, allowing for
pre- to posttesting using a different form. LAS reading and writing scores range from 0 to
100. The numeric results of the tests are grouped into five ranges: 1) low beginner (0 to 20);
2) high beginner (21 to 40); 3) low intermediate (41 to 60); 4) high intermediate (61 to 80); and
5) advanced (81 to 100).
a-ii
Comprehensive Adult Student Assessment System (CASAS): CASAS, San Diego, California. 1994.
The CASAS is a comprehensive curriculum management and assessment system designed
to assess identified competencies of educational programs for all levels of English as a
Second Language. A CASAS appraisal is to be given prior to assessment to determine the
level at which participants should be tested. The Listening test contains 25 to 34 items,
depending on the test level.
The CASAS listening test is administered from tape and generally is individually adminis-tered.
The numeric CASAS scores, which have a potential range of 169 to 228, are grouped
into four levels of functioning: 1) adults scoring below a 200-scaled score are beginners and
have difficulty with basic literacy skills; 2) adults scoring between 200 and 214 can function
in intermediate levels requiring minimal literacy skills; they are able to satisfy basic survival
needs and some limited social demands; 3) adults scoring between 215 and 224 are func-tioning
above a basic level; they are generally able to function in jobs or job training that
involves oral instruction, but usually have difficulty following more complex sets of direc-tions;
and 4) adults scoring at or above 225 can perform work that involves following oral
directions in familiar and unfamiliar situations.
Tests of Adult Basic Education (TABE) 7 & 8. CTB/McGraw Hill Publishing Company, Monterey
California. 1994. TABE 7 & 8 includes subtests in reading, applied mathematics, language,
and spelling. The parallel versions, 7 and 8, allow for pre- to posttesting using different
forms. Tests are available in four skill levels: 1) easy; 2) medium; 3) difficult; and 4) ad-vanced;
and were administered based on participants’ level of functioning.
The TABE can be group or individually administered. The TABE provides grade level
equivalents ranging from 0 (kindergarten) to 12.9 (12th grade, 9 months) and scale scores,
which allows for the accurate measurement of growth in the skills measured. Scale scores on
applied math range from 200 to 795, language ranges from 235 to 826, reading from 160 to
812, and spelling from 220 to 745.
Assessments of Parenting Skills
Parent as a Teacher (PAAT). Scholastic Testing Service, Inc. Bensenville, Illinois.
1984. The PAAT is intended for mothers and fathers with children between three and
nine years of age. This composite attitude scale reveals how individuals feel about
certain aspects of the parent-child interactive system, their standards for assessing the
importance of various child behaviors, and their value preferences concerning child
behavior. The content for PAAT items was derived from a search of the literature re-garding
parental influence upon child development. The PAAT includes five 10-item
subscales: 1) creativity; 2) frustration; 3) control, 4) play; and 5) teaching learning. Sub-scores
range from a low of 10 to a high of 40 with 25 the average, and the point of dif-ferentiation
between desirable and undesirable parental attitudes based on the consen-
a-iii
sus of child development research. On the total score, which ranges from 50 to 200, 125
is the point between desirable and undesirable parenting attitudes.
The PAAT can be administered individually or in a group setting. Spanish and Navajo
versions of the assessment are available. The Spanish version was used extensively in
the evaluation.
Behavior Frequency Assessment (BFA). Office of the Auditor General staff developed
the BFA specifically to measure frequencies of ten specific behaviors related to parental
involvement in their children’s education. Parents were asked to report how frequently
during a week they engaged in the following behaviors during the past week: 1) talk to
their children about what they did that day; 2) read or look at books with their children;
3) read or write in the presence of their children; 4) take their children to the library; 5)
play with their children. Parents with school-aged children were also asked to report
how often during the last 30 days they engaged in the following additional items: 6) talk
to your children’s teacher; 7) go to school activities; 8) volunteer with school activities;
9) help your children with homework; and 10) talk to children about school.
Child Development Assessment
Activity Evaluation System for Observing Preschooler (AESOP). Dowling, J., L.
Wiener, and J. Creighton. Syndactics; Phoenix, AZ. 1994. The AESOP was developed
under contract to the U.S. Department of Education and a prepublication version of the
AESOP has been used for the evaluation.
The AESOP is designed in recognition of the fact that in the early years, a child’s growth
is uneven at different ages and across developmental domains. The assessment provides
multidimensional contexts in which the child has the opportunity to demonstrate
strengths and weaknesses across several domains. It meets the demands required of
systematic observation: it is selective and unobtrusive, and it allows for the observation
of several aspects or components of a child’s behaviors. The AESOP provides raw
scores, which can be interpolated into age-performance scores for a total and three
subscores: 1) problem solving; 2) language; and 3) motor skills.
Family Assessments
The Intake Form was developed by the Office of the Auditor General staff in collabora-tion
with Model Program staff specifically to gather information on the demographics
and background of program participants. Information such as family size, income,
marital status, and previous participation in adult education programs is collected on
the form.
a-iv
The Exit Form was also developed by Auditor General and Model Program staff spe-cifically
to gather information on the services each family received and the outcomes
they achieved. Information such as whether families received transportation and child-care
and employment status at the end of the school year, or when the family exited the
program, is recorded on the form.
Site Assessments
The Site Observation Form was developed specifically by Auditor General staff to
evaluate the services provided by the Family Literacy Pilot Program sites. It was mod-eled
after the validation instruments used by the National Center for Family Literacy
and the National Association for the Education of Young Children.
The Site Observation Form includes items specific to the four major components of
Family Literacy: 1) adult education; 2) parent time; 3) early childhood education; and 4)
parent and child together time. In addition, there is an item specific to program integra-tion.
Items that are observed are rated as “no implementation,” ”partially imple-mented,”
or “fully implemented.”
The Adult and Early Childhood Education Interview Rating Form was designed to
complement the Site Observation Form. It was designed for use during site observation
visits and includes questions for the adult educators, early childhood educators, and
program coordinators.
The Interview Rating Form contains items specific to assessment, student orientation,
retention efforts, curriculum, and integration of program components.
The Contractor Survey supplemented the data recorded on the observation and inter-view
rating forms during the site visits. The survey contains information specific to
schedules, funding, and staffing.
Appendix B
(This Page Intentionally Left Blank)
b-i
APPENDIX B
TESTING PROCEDURES
Teachers Were Provided Training to
Administer Assessments
Family Literacy Pilot Program staff were provided training on how to administer the as-sessment
instruments. Two regional trainings were held in the summer of 1995. Adult
educators were trained by staff from the Arizona Adult Literacy and Technical Resource
Center in the use of the TABE and LAS. As required by the developers, a trainer from
CASAS provided the CASAS training. Staff from the Auditor General’s Office provided
training on the use of the Parent as a Teacher Inventory and on completing the Behavior
Frequencies and Intake and Exit Forms. In addition, one of the AESOP developers provided
training to the early childhood educators on the proper use and scoring of the AESOP.
Despite the efforts to ensure all staff were competent in using the instruments, there is
evidence that some less-than-adequate procedures were used at some sites. Data on some
participants has been excluded from analysis due to observed problems with the data and
some AESOP scores may be deflated due to the testing problems.
Administration
Family Literacy staff were instructed to administer all pretest assessments within 30 days of
participants entering the Program. All posttest assessments were to be administered within
30 days of the end of the Program, or at the time a family leaves the Program if they exited
before the end of the school year.
Dates of administration indicate that site staff complied with the instructions. However,
since many families left the Program without notifying the staff, many families do not have
posttest data.
b-ii
(This Page Intentionally Left Blank)
Appendix C
(This Page Intentionally Left Blank)
c-i
Appendix C
Table 10
Family Literacy Pilot Program
English for Speakers of Other Languages
Participants Matched Pre- to Post-t-test
Language Assessment Scales (LAS) and
Comprehensive Adult Student Assessment System (CASAS) Tests
1995-96 and 1996-97 School Years
Mean
Standard
Score
Standard
Deviation
Mean
Difference
Standard
Deviation
of Mean
Difference t-value Probability
Number
of
Cases
LAS Reading
1995-96 school year
Pretest 52.0694 21.815
10.4440 19.876 4.46 <.001 72
Posttest 62.5139 20.825
1996-97 school year
Pretest 52.2000 22.623
17.4600 18.298 9.54 <.001 100
Posttest 69.6600 16.223
LAS Writing
1995-96 school year
Pretest 31.0154 23.384
12.3077 21.124 4.70 <.001 65
Posttest 43.3231 26.607
1996-97 school year
Pretest 33.2323 24.097
20.8081 20.401 10.15 <.001 99
Posttest 54.0404 21.919
LAS Reading and
Writing
1995-96 school year
Pretest 80.3380 39.386
21.6620 31.899 5.72 <.001 71
Posttest 102.0000 43.315
1996-97 school year
Pretest 82.4242 40.234
39.0808 29.881 13.01 <.001 99
Posttest 121.5051 35.520
(Continued)
c-ii
Appendix C
Table 10
Family Literacy Pilot Program
English for Speakers of Other Languages
Participants Match