ANNUAL EVALUATION
FAMILY LITERACY
PILOT PROGRAM
State o f Arizona
office
o f the-
Auditor General
Report to the Arizona Legislature
By Douglas R. Norton
Auditor General
~ ecember1 996
Report 96- 20
DOUGLAS R. NORTON, CPA
AUDITOR GENERAL
STATE OF ARIZONA
OFFICE OF THE
AUDITOR GENERAL
DEBRA K. DAVENPORT, CPA
DEPUTY AUDITOR GENERAL
December 31,1996
Members of the Arizona Legislature
The Honorable Fife Symington, Governor
The Honorable Lisa Graham Keegan
State Superintendent of Public Instruction
Transmitted herewith is a report of the Auditor General, an Annual Evaluation of the Family
Literacy Pilot Program. This report is in response to the provisions of Session Laws 1994,
Ninth S. S., Chapter 1, 59.
Ths is the second in a series of three reports. The final evaluation is scheduled to be released
on or before December 31,1997. Our evaluation finds that during the first year of program
implementation, many contractors did not meet criteria required by Arizona's Family Literacy
Pilot Program. Also, whle ADE did close two sites due to their failure to meet contractual
obligations, ADE's monitoring of most sites during the 1995- 96 school year was inadequate.
Continued inadequate monitoring may result in contractors' ongoing lack of compliance with
program requirements. In addition, due to budgeting errors that were made in the past, some
sites are operating with inadequate resources, whch may result in programs not being able
to produce the desired outcomes. Finally, the Program is having problems enrolling and
retaining participants, which may h i t the outcomes evaluation that is due in the final report.
My staff will be pleased to discuss or clarify items in the report.
This report will be released to the public on January 3,1997.
Sincerely,
"~ ou~ las R. Norton
Auditor General
I 2910 NORTH 44TH STREET m SUITE 410 m PHOENIX, ARIZONA 85018 . ( 602) 553- 0333 . FAX ( 602) 553- 0051
SUMMARY
The Office of the Auditor General has completed the second year of a three- year evaluation
of the Family Literacy Pilot Program. This evaluation was conducted pursuant to the provisions
of Laws 1994, Ninth S. S., Ch. 1, 59. ' Ihs second- year interim evaluation report provides
information regarding the Program's first full year of implementation and recommendations
regarding the Program. The final evaluation report will focus on the Program's impact and
is to be released on or before December 31,1997.
The Legslature established the Family Literacy Pilot Program through legislation known as
the Chddren and Family Stabihty Act of 1994. The Program is administered by the State Board
of Education through the Arizona Department of Education ( ADE) Division of Adult Education.
ADE contracts with local school districts and other community and educational organizations
to provide program services.
Arizona's Family Literacy Pilot Program strives to end and prevent the intergenerational cycle
of undereducation and poverty by improving the basic literacy skills of eligible parents and
their preschool children. Arizona's Family Literacy Pilot Program uses a model that integrates
components advocated by the National Center for Family Literacy ( NCFL). Four components
( adult literacy instruction, early chldhood instruction, parent and child interaction time, and
parent education discussion groups) are integrated to provide comprehensive literacy services.
Lack of Compliance
with Program Criteria
Threaten Potential for Success
( See pages 5 through 11)
During the first year of implementation, many contractors did not meet criteria required by
Arizona's Family Literacy Pilot Program. The framework in whch Family Literacy Pilot
Programs must operate is clearly defined by the 1994 legislation that created the Program.
Under this legislation, contractors are required to enroll an established number of families who
meet specific eligibility criteria, provide and integrate the four major components of a family
literacy program, collaborate with other agencies, and provide for families' child care and
transportation needs. Sites are also contractually required to provide a minimum number of
service hours to families.
However, many sites did not meet statutory or contractual criteria. Only 7 of 26 family literacy
sites met all requirements. Two of the 26 sites were closed in the second half of the 1995- 96
school year due to their failure to provide contracted services. The other 17 sites did not fulfill
between 1 and 4 requirements.
The Program was originally structured to comply with nationally recommended guidelines
for service delivery, and sites must follow these national p d e h e s to acheve their full potential
for success. Sites' noncompliance with the statutory and contractual requirements, and sites
not complying with NCFL guidehnes, increase the Program's cost per family and undermine
its intent.
ADE Should Conduct
Comprehensive
Program Monitoring
( See pages 13 through 15)
Even though monitoring is critical to ensure sites are adhering to the program criteria, ADE
does not do an adequate job in this area. Although ADE did close two sites due to their
repeated fdure to meet contractual obligations, ADE's monitoring of most sites during school
year 1995- 96 was inadequate. Most program sites received little monitoring by ADE. The few
site visits ADE conducted were not formally structured, nor did it provide formal feedback
to contractors regarding deficiencies. The monitoring process should involve on- site contact,
structured classroom observation, budget and expenditure reviews, checks for compliance with
legslated program requirements, program record and documentation reviews, and follow- up
when specific deficiencies are identified.
If ADE continues to inadequately monitor the Program, compliance problems are likely to
persist ADE should improve its monitoring by committing more resources and increasing staff
expertise by collaborating with other entities who have more experience with family literacy,
such as the two Family Literacy model programs that are contractually responsible for
providing techrucal assistance to the pilot sites and the federal Even Start Program, which is
also administered by ADE.
Past Budgeting Errors Continue
to Negatively Affect Sites
( See pages 17 through 19)
In school years 1994- 95 and 1995- 96, ADE awarded more monies to contractors than were
available. Those budget overallocations created the need for a mid- year budget cut for 1995- 96
and an additional reduction for the 1996- 97 school year. These cuts have severely threatened
some contractors' ability to provide quality programs with adequate resources. While many
of the sites have financial support from sources other than the State, sites without enough
monies to provide all of the services included in the model may not produce the desired
outcomes.
In addition, the programs were supported financially at lower levels than what is recommended
by NCFL, and the 1996- 97 and 1997- 98 appropriations are able to support only 16 to 17 sites
at recommended dollar levels, rather than the current 24. Since inadequate support of the sites
is likely to reduce program effectiveness in achieving the outcomes and goals, if the Program
is continued in the future ADE should require contractors to have program funding plans to
ensure sites have adequate resources.
Statutory Annual
Evaluation Components
( See pages 21 through 29)
This report also contains information required to be included in each annual evaluation by Laws
1994, Ninth S. S., Ch. 1/ 59. As part of this information, we note that due to the difficulty the
Program is having enrolling and retaining participants, the outcomes evaluation due next year
may be limited. Only a small number of families will have been assessed on the outcome
measures at least two times.
iii
( This Page Intentionally Left Blank)
Table of Contents
Introduction and Background . . . . . . . . . . . . . . . . . . . . . . . . .
Finding I: Lack of Compliance with
Program Criteria Threaten
Potential for Success . . . . . . . . . . . . . . . . . . . . . . . . . . .
Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Programs Fail to
Comply with Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contractors' Experience
Explains Variations in
ProgramQuality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Impact of
Noncompliance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Finding II: ADE Should
Conduct Comprehensive
Program Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
ADE's Monitoring Process
ContinuestoBeInadequate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Adult Education Division
Should Collaborate with Model
ProgramsandEvenStart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Recommendation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Paae
1
Table of Contents ( con't)
Page
Finding Ill: Past Budgeting
Errors Continue to
Negatively Affect Sites . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Family Literacy
ProgramFunding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
ADE Reduced Contractors'
Funding to Correct Budgeting Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Programs Affected
byBudgetReductions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
ADE Should Require
Contractors to Have
aFundingPlan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Recommendation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Statutory Annual
Evaluation Components . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Agency Response
AppendixA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . a- i
Table of Contents ( concl'd)
Page
Tables
Table 1: Arizona Family Literacy Pilot Program
Instances of Noncompliance by Site and Requirement
Year Ended June 30,1996 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Table 2: Arizona Family Literacy Pilot Program
Average and Total Number of Families Served
at Each Family Literacy Site
Year Ended June 30,1996 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Table 3: Arizona Family Literacy Pilot Program
Contractors and Contract Amounts
Years Ended or Ending June 30,1995,
1996, and 1997
( Unaudited) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Table 4: Arizona Family Literacy Pilot Program
State Cost per Family per Site
Year Ended June 30,1996
( Unaudited) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Table 5: Arizona Family Literacy Pilot Program
Costs of Comparable Programs . . . . . . . . . . . . . . . . . . . . . . . . . 27
vii
( This Page Intentionally Left Blank)
INTRODUCTION AND BACKGROUND
The Office of the Auditor General has completed the second year of a three- year evaluation
of the Family Literacy Pilot Program. This evaluation was conducted pursuant to the provisions
of Laws 1994, Ninth S. S., Ch. 1, 59. This second- year interim evaluation report provides
information and recommendations regarding the Program. The final evaluation report will
focus on the Program's impact and is to be released on or before December 31,1997.
Family Literacy in Arizona
The Legslature established the Family Literacy Pilot Program with legislation known as the
Children and Families Stability Act of 1994. The Program's intent is to increase the basic
academic and literacy skills of undereducated parents of preschool chldren.
Appropriations for the Family Literacy Pilot Program were $ 975,000 and $ 1,000,600 for school
years 1994- 95 and 1995- 1996, respectively. The Legislature appropriated $ 1,000,000 for both
the 1996- 1997 and 1997- 1998 school years.
Anzona is one of four states that currently support family literacy programs. The other three
states are Kentucky, South Carolina, and Washington. Arizona began committing monies to
support the Family hteracy Pilot Program in 1994. The Program is administered by the State
Board of Education through the Division of Adult Education, Arizona Department of Education
( ADE). ADE contracts with other organizations to provide program services.
Contractors for 1995- 96 include 10 school districts, 3 community- based organizations, 1 college,
and 1 adult education provider. Most sites serve urban areas, with 19 of the 26 sites serving
metropolitan Phoenix, Tucson, and Yuma. Seven sites serve Arizona's rural population, with
two sites in Cochse and Coconino Counties, and one site each in Apache, Navajo, and Yavapai
Counties. Two model programs, Mesa Family Tree and Pima County Adult Education, provide
technical assistance to all sites and training to family literacy staff. Because the Chldren and
Famhes Stability Act did not become effective until after the start of the 1994- 95 school year,
and the need to award the Family Literacy grants through a Request for Proposal process,
contractors generally did not begin to provide services to famihes until the 1995- 96 school year.
As mandated by the Legislature, family literacy serves preschool- aged children and their
parents in areas where many people live in poverty, and have low levels of education. Eligible
parents include individuals who: 1) have a three- or four- year- old child; 2) lack sufficient
mastery of basic educational or basic English language skills needed to function in society, or
lack a high school diploma or its equivalent; and 3) are U. S. or legal residents, or are otherwise
lawfully present in ths country.
Program Goals and Services
Arizona's Family Literacy Pilot Program strives to end and prevent the intergenerational cycle
of undereducation and poverty by improving the basic literacy slulls of eligible parents and
their preschool children. To accomplish these goals, the Legislature mandated that contractors
provide the following basic services:
Identify and recruit eligible parents and children.
Screen and prepare parents and children for participation in the Program.
Serve at least 10 but no more than 15 parents with children eligible for the Program.
Provide instructional programs that promote academic and literacy slulls and equip parents
to provide needed support for their children's educational growth and success.
To encourage families to fully participate, contractors must have a plan for the following
auxiliary services:
Provision of food services for program participants.
Provision of chld care through either private or public providers.
Transportation for participants.
Additionally, two requirements that structure the way the Program is offered are:
Collaborate in organizational partnerships involving at a minimum a common school, a
preschool provider, and an adult education program paid for by the Department of
Education.
Use a national family literacy model.
Program Model
Arizona's Family Literacy Pilot Program uses a model that integrates components advocated
by the National Center for Family Literacy ( NCFL). Adult literacy instruction, early childhood
instruction, parent and child interaction time ( PACT), and parent education discussion groups
are integrated to provide comprehensive literacy services.
Adult literacy instruction includes either adult basic education ( ABE) and general
educational development ( GED), or English for Speakers of Other Languages ( ESOL)
instruction. Based on their individual needs, eligible parents receive either the ABE/ GED
or ESOL component.
Early childhood instruction uses a developmentally appropriate curriculum to foster early
literacy skills. Teachers focus on developing the cognitive, physical, social, and emotional
skills critical to future academic success.
1 Parent and child interaction time ( PACT) allows parents and their children to work and
play together as a functional family unit. Chld- initiated activities allow parents to learn,
teach, and communicate with their children, using positive parenting skills that are
transferable to the home.
Parent education discussion and support groups make it easier for parents to discuss
issues or problems surrounding p'arenting. Participants learn from the experiences of their
peers, receive encouragement from the group, and practice collective problem solving.
Component integration is key to successful family literacy, and can occur when staff
organize activities that involve all components and create links between them.
An example of component integration at one site related to safety. Because of the violence in
the neighborhood, the parents felt anxiety about their preschoolers being on the playground.
The parents talked about keeping the chldren inside all of the time because of the threat of
violence. In Parent Time, the parents developed action plans to address the threat of violence.
The action plans included finding a book on keeping children safe and reading it, meeting with
the school principal about safety in the school, and inviting a speaker to their Parent Time to
talk about gangs and how the parents could talk to their children about gangs. Sections of the
book on chddren's safety were incorporated into the Adult Education time. Parents also took
the book home and shared it with other family members. In parent time, the parents made up
a song about keeping safe, especially around strangers, which they then taught their chldren
in PACT. Parents made books for their children about situations that had caused the parents
to be fearful. Also during PACT, parents would tell stories about how their children used a
safety rule and the children received a safety award. In the Early Chldhood class, the children
were introduced to a new puppet, " Buzzy, the Safety Bee," who gave the children safety
reminders.
Scope and Methodology
The Family Stability Act requires the Office of the Auditor General to conduct three annual
evaluations on the Family Literacy Pilot Program. Methods used in tlus evaluation include
interviews with family literacy administrators and program staff, structured observation of
25 of 26 program sites, 4 home visits, analysis of data on families enrolled in the Program, and
review of comparable programs. Appendix A ( see page a- i), provides a discussion of the
methods used to collect information on the Arizona Family Literacy Pilot Program.
Twenty- five of the 26 family literacy sites were visited for structured observation and to conduct
staff surveys? Observation forms were used to record information on each of the family literacy
components. Interviews were conducted with staff to gather information on their qualhcations,
problems encountered, and plans for improving the Program's operation. Visits for program
evaluation purposes were conducted after January 1,1996, to allow contractors time to initiate
activities. All visits were completed by June 30,1996.
In the first year it was reported that the program model used by the Arizona Family Literacy
Pilot Program followed nationally recommended standards. Budgeting errors that created a
money shortage for second- year services were discussed, and it was recommended that family
literacy administrators establish proper budgeting procedures and consider the availability
of additional local, state, and federal dollars when making funding decisions. In addition, it
was recommended that ADE expand and enhance program monitoring by family literacy
administrators, and develop a structured and coordinated monitoring plan to ensure adequate
expertise exists to provide program oversight.
In this second- year report, we again address problems identified in the first year, and
Provide an overview of sites' failure to comply with state and national guidelines;
Review ADE's failure to provide adequate monitoring for compliance with the program
model; and
Discuss continuing problems caused by the first- year budgeting error.
The Auditor General and staff express appreciation to the Superintendent of Public Instruction,
Adult Education Division Director, Family Literacy Coordinator, and staff of the Arizona
Department of Education's Adult Education Division, as well as the Family Literacy Program's
staff and families for their cooperation and assistance during the second year of the Family
Literacy Pilot Program evaluation.
1 One site was not visited due to its failure to agree to a site visit time. This site has been subsequently closed by
ADE and the DOA because it failed to provide contracted services.
FINDING I
LACK OF COMPLIANCE WITH
PROGRAM CRITERIA THREATEN
POTENTIAL FOR SUCCESS
Many of the Arizona Family Literacy Pilot Program sites did not comply with the program
model, which threatens the Program's potential effectiveness. Although specific program criteria
are clearly defined, most of the sites did not meet at least one of them. Sites' success in meeting
the criteria depends directly on contractors' experience with operating family literacy programs.
Programs not meeting criteria may decrease the Program's overall effect on families.
Background
As reported in the first evaluation report on the Family Literacy Pilot Program ( Auditor General
Report 95- 20), sites have strong potential for success in achieving overall goals and objectives.
Sites were originally structured to comply with nationally recommended guidelines for service
delivery; however, they must continue to follow the national guidelines if they are to acheve
their full potential for success. Contractors must comply with guidelines established by the
National Center for Family Literacy ( NCFL), a non- profit organization committed to the
establishment of Family Literacy programs. The Program's enabling legislation specifically
requires the model developed by NCFL be used by the contractor unless a model deemed
superior by the contractor and the State Board of Education is used.
Programs Fail to
Comply with Criteria
During the first year of program implementation many contractors did not meet criteria
required by Arizona's Family Literacy Pilot Program. The framework in which family literacy
programs must operate is clearly defined. The Children and Family Stability Act of 1994
established specific criteria for family literacy providers to meet. Contractors are required to
enroll an established number of families, enroll only families who meet specific eligibility
criteria, provide and integrate the four major components of a family literacy program,
collaborate with other agencies, and meet families' child care and transportation needs. Sites
are also contractually required to provide a minimum number of service hours to families.
However, many sites faded to comply with statutory or contractual criteria. Only 7 of 26 family
literacy sites are in compliance with all requirements. Two of the 26 sites were closed in the
second half of the 1995- 96 school year because they failed to provide contracted services, and
the other 17 sites did not fulfill between 1 and 4 requirements. Table 1 ( see pages 7 through
8), lists sites and their areas of noncompliance.
Enrollment- Only 12 sites served the statutorily required number of 10 to 15 families.
Contractors are required to enroll at least 10 but no more than 15 parents with children eligible
for the Program. Table 2 ( see page 9), shows a distribution of the average number of families
served per month and the total number of families served throughout the 1995- 96 school year
for each site. Sites that served 10 to 15 families for at least 7 months of the average 9- month
school year met the enrollment criteria.
Eligibility - Two sites served ineligble participants. These two sites enrolled at least one family
that either had a child older than four years at time of enrollment, the adult already had a GED,
or the adult is not a legal reGdent of the U. S. At one site, several adults not only had a hgh
school diploma or GED, but had also attended several semesters of college.
National Center fm Family Litera y ( IVCFL) model- Five sites did not implement either the
parent and child interaction time component or the parent discussion and support group
component All sites provided the adult literacy instruction and early childhood instruction.
Some sites had parent time or parent and chld interaction scheduled, but did not actually do
the activities or the activities did not qualify as the component. Some sites may have added
the components after receiving techrucal assistance from one of the model programs. The NCFL
recommends that all four components be integrated, because each is essential for the Family
Literacy Program's effectiveness.
Integrated components- Seven sites did not integrate the four family literacy program
components well. As noted previously, it is the integration of the four components that is
considered the key to the Program's success. Integration was determined by observations of
program components and interviews with program staff. The extent to whch sites integrated
components may have improved as a result of techrucal assistance provided by the model
program after the sites were first observed.
Collabmntion- One of the sites had poor collaboration with other services.' Contractors are
required to maintain an organizational partnershp involving at a minimum a common school,
a private preschool provider, and an adult education program paid for by the ADE. The site
did not collaborate with an established preschool provider.
Child care- Four sites failed to meet the child care needs of younger siblings of chldren in
the programs. Contractors are required to have a plan for child care through either a public
or private provider. Lack of ths service ultimately affects parents' attendance if they cannot
afford a baby- sitter or do not have anyone to watch their infants while they are in class.
' Sites in remote areas that are unable to collaborate with other services because potential collaborators do not
exist are not considered to be out of compliance.
Table 1
Arizona Family Literacy Pilot Program
Instances of Noncompliance by Site and Requirement
Year Ended June 30,1996
NCFL Com- ' Components
Enrollment Eligibility ponents Integration Collaboration Child Care Transportation
Service
Site Hours Total
Crane ESD No. 13
Crane Morning Session
Crane Afternoon Session
Flagstaff USD No. 1
Killip
~ P P
Isaac ESD No. 5
Isaac English for Speakers
of Other Languages
Isaac GED
Mesa USD No. 4
Ebenhower
Lincoln
Longfellow
Pima County Adult Education
Liberty
Nash
Ochoa
Pinon USD No. 4
Red Mesa USD No. 27
Somerton ESD No. 11
Tempe ESD No. 3
Tucson USD No. 1
Iawrence
MeyewGanoung
Pueblo Gardens
Tucson High School
Wakefield
Table 1 ( con't)
Arizona Family Literacy Pilot Program
Instances of Noncompliance by Site and Requirement
Year Ended June 30,1996
Site
Other
NCFL Corn- Components Service
Enrollment Eligibility ponents Integration Collaboration Child Care Transportation Hours Total
Cochise Community College X X X X
Literacy Volunteers of
Maricopa County X X X X
Phoenix Indian Center X
Total -- 12 - 2 3- 2 2- 2
cn
Note: Chino Valley USD No. 51 and the Council for Family Concern in Willcox were closed and are not included in the table.
Source: Auditor General staff analysis of data provided by the Arizona Department of Education and Family Literacy Program contractors.
Table 2
Arizona Family Literacy Pilot Program
Average and Total Number of Families Sewed
at Each Family Lieracy Site
Year Ended June 30,1996
Site
Crane ESD No. 13
Crane Morning Session
Crane AfternoonSession
Flagstaff USD No. 1
Killip
L~ UPP
Isaac ESD No. 5
Isaac English for Speaker of Other Languages
Isaac GED preparation
Mesa USD No. 4
Eisenhower
Lincoln
Longfellow
Pima County Adult Education
Liberty
Nash
Ochoa
TucsonUSD No. 1
Lawrence
MeyersGanoung
Pueblo Gardens
Tucson High School
Wakefield
Contractors with single classroom sites
Cochise Community College
Literacy Volunteers of Maricopa County
Phoenix Indian Center
Pinon USD No. 4
Red Mesa USD No. 27
SomertonESD No. 11
Tempe ESD No. 3
Total
Average Number
of Families
Per Month
Total Number
of Families
Served
Note: Chino Valley USD No. 51 and Council for Family Concern in Willcox were closed and are not included in this table.
Source: Audtor General staff analysis of data provided by Family Literacy Program contractors.
Tmnsportntion- Two sites did not provide an adequate transportation plan for families to
attend class, as they are required to do. For example, one site was in an isolated area that Backed
public transportation. Site staff proposed transportation arrangements on an as- needed basis
for families; however, as the school year progressed, solutions such as car pooling became an
unreliable source of transportation because of people dropping out of the Program or having
car problems. Other transportation options were not available, which resulted in participants
leaving the Program.
Service hours- Five sites failed to provide participants with the minimum number of 15 service
hours per week. Parents received fewer than 10 hours of service at one site. Both children and
parents attended 12 hours at another site. All sites counted breakfast or lunch as service hours.
According to NCFL recommendations, these hours do not count as service hours unless adults
participate in family- style meals with children and the families are supervised by an instructor,
The meal hours can be counted as part of the adult curriculum only if supervised instruction
( either formal or informal) is provided during this time.
Contractors' Experience
Explains Variations in
Program Quality
Contractors' prior experience with family literacy programs affect how well the individual sites
implement program criteria. More experienced contractors provide higher- quality programs
that typically meet the Program's criteria.
Sites with experienced staff- Sites with experienced staff are hgh in overall quality and are
more likely to meet a hgher number of program criteria. Contractors who have conducted
family literacy programs such as Even Start for more than five years provided consistently good
programs at their family literacy sites. These sites integrate core program components according
to national standards to offer quality service to participants and establish strong collaborations
with other available agencies. Experienced sites maintain high enrollment by implementing
attendance contracts with consequences for failure to attend, and conducting home visits. At
these sites, parents agree to attend on a regular basis. Certificates are given to those with perfect
attendance and warnings are given to parents with irregular attendance that they will be
terminated from the Program if poor attendance continues. Participants in such programs are
aware that it is costly to operate the Program and regular attendance is impo~- tantA dditionally,
staff at experienced sites document and follow up on participants' literacy objectives to
accomplish program goals.
New sites- Sites in their first year of providing family literacy programs show more variation
and lower overall quality than more established sites. Sites operated by providers with no prior
family literacy experience have trouble integrating family literacy components. These sites also
have not yet been able to establish collaborative relationships with other agencies to provide
speclfic family literacy components. For example, one site did not collaborate with an existing
preschool provider. As a result, the newly created and independently run early chldhood
classroom provided poorquality services and lacked the appropriate curriculum for chldren.
These sites tend to focus on the instruction for individual family members and neglect the
important family components, such as the parent and child activities, that make the literacy
program unique.
Impact of
Noncompliance
Sites not complying with the statutory and contractual requirements and NCFL standards affect
the Program's cost per family and undermine its intent.
High cost per family- Contractors' lack of compliance with enrollment requirements increased
the Program's costs per famdy. For school year 1995- 96, the average cost per family was $ 5,198.
If all sites served the maximum number of 15 families throughout the year, this amount would
be reduced, to $ 3,229. Costs per family are further discussed in the Statutory Annual Evaluation
Components section ( see pages 21 through 29).
Lack of essential services jeopardize positive outcmsfoufamilies - Sites that do not provide
child care, a viable transportation plan, or lack sufficient number of adult instruction service
hours make it more difficult for participants to aclueve the desired program outcomes. When
these requirements are not met and site staff make minimal efforts to keep families in the
Program, famhes' attendance is negatively affected. The NCFL recommends that site staff make
efforts to maintain famhes' attendance on a regular basis for at least a year. Research suggests
that regular attendance and participation is needed for programs to make lasting impacts on
families.
Recommendations
1. ADE should work with its family literacy contractors to ensure they take the following
actions to comply with the program model:
Implement and integrate the four family literacy components consistently
Increase retention efforts to maintain 10 to 15 families at all times
Collaborate with other organizations
Conduct regular home visits that are consistent and purposeful.
( This Page Intentionally Left Blank)
FINDING II
ADE SHOULD CONDUCT
COMPREHENSIVE PROGRAM MONITORING
ADE does not adequately monitor programs. Last yeafs report discussed inadequacies with
ADE's monitoring processes and lunited commitment of resources devoted to comprehensive
oversight for Family Literacy Pilot Programs. Although ADE did close two sites because they
repeatedly failed to meet contractual obligations, ADE's monitoring of most sites during school
year 1995- 96 was inadequate. Monitoring is critical to ensure sites are adhering to the program
criteria. Without adequate monitoring, compliance problems will persist and poor- quality
services will continue. ADE should improve its monitoring by committing more resources and
increasing staff expertise by collaborating with other entities.
Background
Last yeds report encouraged ADE to expand its monitoring of Arizona's Family Literacy Pilot
Program to ensure that sites are providing quality services. The report also focused on how
ADE's lunited staff resources and expertise would jeopardize its ability to adequately monitor
the Program. Recommendations to develop a monitoring plan in partnership with the model
programs and coordinating site visits with the models and other ADE staff were also made.
ADE's Monitoring Process
Continues to Be Inadequate
Although ADE closed two sites during 1995- 96, its monitoring continues to be insufficient for
program oversight. ADE has conducted few visits to family literacy sites and lacked a
structured process to rate or document sites' quality. ADE did not commit sufficient resources
to monitoring nor did it improve its in- house expertise.
ADE closed two sites for poor p@ omzance- ADE closed the sites operated by Chno Valley
Unified School District and the Council for Family Concern in Willcox because the sites failed
to provide contracted services. Both sites had severe problems that were detectable by even
limited monitoring. These sites failed to submit program reports and data and were unable
to demonstrate that they were providing services. Even when ADE scheduled visits to the sites,
site staff faded to have parents and chldren in attendance during the visits. In addition, ADE
used information that the Auditor General's Office and the model programs provided about
problems observed during site visits in making the recommendation to close the sites.
Most propm sites & ved little monitoring by ADE- The few site visits conducted by ADE
were not formally structuwd, nor did the Department provide formal feedback to contractors
regarding deficiencies. Site visits should involve on- site contact, structured classroom
observation, budget and expenditure reviews, checks for compliance with legislated program
requirements, program record and documentation reviews, and follow- up when specific
deficiencies are identified.
ADE does not have adequate resources to monitorfamily litera y- ADE staff are unable to
dedicate sufficient attention to the Program. Family Literacy Pilot Program monies no longer
support the staff member overseeing the Program; instead, the Family Literacy Pilot Program
coordinator is supported by other state and federal monies and now spends 40 percent of her
time on other programs. While ADE origmally devoted about 5 percent of the Family Literacy
Pilot Program appropriation for agency administration, including monitoring, due to budgeting
problems ( see Finding III, pages 17 through 19), ADE now uses the entire appropriation to
support direct program activities.
The Division needs to improve expertise in vnonitoving family litera y - Although our first-year
report noted the need to enhance the program monitoring process by having Family
Literacy administrators collaborate with model programs and other ADE divisions, the
Division did not make efforts to collaborate with other resources to increase staff knowledge
of family literacy and thus, improve monitoring. Since there were limited monies to support
monitoring, collaboration with model programs or other ADE- administered early chldhood
programs would have improved the monitoring process.
Adult Education Division
Should Collaborate with Model
Programs and Even Start
ADE could address the need for improved monitoring by drawing upon resources and
expertise available from other sources. Last yeafs report recommended that ADE collaborate
with the model programs and programs such as Arizona's federally supported Even Start
Program, whch is also operated through ADE.' ADE could improve its program monitoring
skills by collaborating with other programs.
Model sites could provide assistance in propm monitoring- Currently, as part of the Family
Literacy Pilot Program, two model programs, Mesa Unified School District's Family Tree Project
and Pima County Adult Education, serve as models and training resources to pilot programs
by providing technical assistance to sites and training to family literacy staff. Trainers from
the model sites conduct visits to family literacy contractors to observe operations and to provide
constructive feedback. While the model programs have frequent interaction with the pilot sites,
both the model programs and ADE do not view monitoring for compliance with statutes as
1 Even Start supports family literacy programs at eight school districts in Arizona.
14
part of the model programs' role. However, the frequency of the model sites' visits to the other
sites and their knowledge of family literacy services qualifies them to report serious
inadequacies in program delivery. At a minimum, informing ADE of these problems should
be part of the model programs' responsibhties. ADE should then take responsibility for further
investigating site problems.
ADE's feh1Ey supported Even Start can serve as an important uesom- The Program could
also draw on expertise already existing within ADE for help in monitoring. Even Start is a
federally supported family literacy program administered by ADE's Title 1 Office. The Even
Start Coordinator is active in monitoring districts for contract compliance and program quality
using a process that includes activities such as conducting informal meetings once a month with
program coordinators for problem solving and networking, and conducting one formal site
visit per year to each site. The site visits use NCFL and federal Even Start p d e h e s and consist
of observations and interviews with staff, parents, and collaborators. Sites are rated in the first-year
evaluation with a structured evaluation form from NCFL. Evaluation for subsequent years
focuses on following up specific issues. Each visit also includes an exit meeting to
provide feedback. If recommendations are made, the site is asked to provide a written plan
describing changes that will be made to meet requirements. Finally, the site visit and
recommendations are formally documented. Several days are scheduled for site visits at each
district to allow for comprehensive program observations and staff interviews.
The Family Literacy Pilot Program could use Even Start materials and adapt their process to
monitor Family Literacy Pilot Program sites. Additionally, the Even Start coordinator could
train Family Literacy Pilot Program staff to conduct site visits. Such training could include joint
visits of sites that operate both Even Start and Family Literacy Pilot Programs, or having the
Family Literacy Pilot Program coordinator accompany the Even Start coordinator on a visit.
The Even Start coordinator could also conduct site visits for the Family Literacy Pilot Program
using the Even Start process.
Recommendation
1. ADE should actively monitor sites for compliance with program requirements and quality
of services. Family literacy administrators should collaborate with model programs and
other ADE divisions to develop a monitoring plan and conduct site visits. Collaboration
with individuals from Even Start and early childhood education programs would provide
for comprehensive oversight of all family literacy components.
( This Page Intentionally Left Blank)
FINDING Ill
PAST BUDGETING ERRORS CONTINUE
TO NEGATIVELY AFFECT SITES
In school years 1994- 95 and 1995- 96, ADE awarded more monies to contractors than were
available. Those budget overallocations created the need for a mid- year budget cut for 1995- 96
and an additional reduction for the 1996- 97 school year. These cuts have severely threatened
the ability of some contractors to provide quality programs with adequate resources;
consequently, in the future ADE should require contractors to have a funding plan
demonstrating adequate resources if sites are to receive state support.
Family Literacy
Program Funding
Family literacy is based on the premise that a program incorporating adult, chld, and parent
education is more effective than dealing with these components independently. However, a
minimum level of resources is needed to provide a quahty program. Although budgets required
to operate a family literacy program vary depending on the number of days per week the
program operates, the program size, and cost of conducting the program in a particular
community, NCFL estimates one full- time program requiws a budget of $ 50,000 to $ 90,000 per
year. Whle many of the sites have financial support from sources other than the State, sites
that do not have sufficient monies to provide all the services included in the model may not
produce the desired outcomes.
ADE Reduced Contractors'
Funding to Correct Budgeting Errors
As reported in our first- year evaluation, ADE awarded more monies to programs than were
avdable in both school years 1994- 95 and 1995- 96. ADE later cut contractors' budgets by 12
percent midway through school year 1995- 96 to correct the overallocations. Contractors that
included established adult and/ or early childhood education providers were able to operate
with decreased budgets, and managed to reallocate their remaining monies to provide services
for the rest of the school year. In contrast, the sites that depended almost exclusively on state
Family Literacy dollars struggled to operate with the decreased amounts. These sites depended
heavily on the state monies for teacher salaries and operating expenses.
In addition to the mid- year budget cuts programs suffered in school year 1995- 96, ADE then
reduced monies by 24.5 percent for all contractors for school year 1996- 97. ADE needed to
adjust awards to equal the $ 1,000,000 appropriated for school year 1996- 97. This resulted in
some sites receiving as little as $ 37,632 for 1995- 96, and only $ 28,412 for 1996- 97. These amounts
are well below the minimum needed to provide all the services called for in the program model.
Programs Affected
by Budget Reductions
The budget decreases of the last two years forced sites to modify services. Modifications
included contractors ending the program year early, reducing staff salaries, cutting budgets
for national staff training, reducing the number of field trips scheduled, and canceling orders
for educational supplies. Reducing staff salaries sometimes resulted in staff turnover that
negatively affected participants' attendance.
The budget cuts affected components critical to the Program's success. For example, some sites
not needing monies for child care at the time budgets were cut ehinated it from their budgets.
However, when need for the monies arose later, they were not available. The lack of chld care
prevented parents who could not afford baby- sitters from attending class. Consequently, fewer
participants attended and more dropped out, which caused sites to be in noncompliance with
enrollment requirements when the budget was cut. Sites also eliminated efforts to recruit
students so that they could maintain the required 10 to 15 families.
ADE Should Require
Contractors to Have
a Funding Plan
In any future awards of family literacy funds, ADE should require Family Literacy contractors
to have a funding plan, clearly demonstrating a site will have adequate resources, in order to
receive state support At the time contractors initially submitted Family Literacy Pilot Program
proposals in response to ADE's Request for Proposal, they were not required to show other
sources or amounts of funding. As a result, ADE was unable to determine if programs had
adequate resources to operate, and some grants were made to contractors that had insufficient
support for their family literacy site. By requiring a funding plan, the ADE can move toward
ensuring that state funds will only be used by sites that have adequate overall funding to
support a program.
I Recommendation
1. In any future awards of family literacy funds, the ADE should require contractors to have
a funding plan demonstrating that each site has adequate overall funding that will allow
I it to operate at a level of quality that has the potential to produce the desired family learning
outcomes.
( This Page Intentionally Left Blank)
STATUTORY ANNUAL
EVALUATION COMPONENTS
Pursuant to Laws 1994, Ninth S. S., Chapter 1, 59, the Office of the Auditor General is required
to include the following information in the annual program evaluation.
1. Information on the number and characteristics of the program participants.
The participant information that follows is presented in composite from all 26 sites.
Prograrn Farnilies - Typically, th6 family members directly served by the Family Literacy
Pilot Program include a mother and her three- or four- year- old child. Occasionally a mother
may have more than one chld in the preschool component ( a three- and a four- year- old,
or twins or triplets). In addition, there are some cases in which a grandparent, aunt, or father
is sewed by the adult education component There are also several cases of more than one
adult from the family enrolled in the adult education component.
Program Parents- Most family literacy adult participants are mothers, though there are
a few who are fathers, a grandparent, or an extended family member. Participants' ages
ranged between 17 and 69. Over 80 percent of adult program participants are 35 years old
or younger.
Progra~ nC hiIdm- The family literacy sites served 77 three- year- old and 244 four- year- old
children. About 60 percent of the families have two or three children in the household.
Number of Enrolled Participants - Throughout the 1995- 1996 school year, sites enrolled
326 participants. By statute, family literacy sites target between 10 and 15 families per site.
Sites enrolled participants continuously throughout the year as enrollment fluctuated due
to attrition and the fad that some class space did not become avadable untd later in the year.
The reported participants did not necessarily complete a full year in the Program. The data
reflects participants' status at enrollment.
Family Ethnicity- The majority of families enrolled are Hispanic ( 76 percent). Other
ethnicities represented are Native American ( 16 percent), White non- Hispanics ( 6 percent),
African- American ( 1 percent), and Asian/ Pacific Islanders ( 1 percent). English language
literacy is low. Only 22 percent of the families reported English as the primary language
spoken at home.
Famihj Status- Participants come from poor and undereducated backgrounds. Almost 60
percent of families have incomes less than $ 10,000, and 81 percent of adult participants have
less than a twelfth- grade education.
Whle 60 percent of participants' spouses ( mostly males) work and provide the primary
source of income for the family, only 12 percent of adult participants are employed. Most
families subsist on one adulfs income, while some others ( 29 percent), also receive some
form of government assistance.
Sixty- four percent of families report their family status as couples with children. Only 23
percent of participants reported themselves as a single parent, whle the remaining 13
percent are extended families caring for children. Overall, sites are serving structurally intact
families who meet the eligibility criteria and aspire to improve their literacy skills and
reduce poverty.
2. Information on contractors and program service providers.
Ftfteen Pilot Propm CmtmAms- The State Board of Education approved family literacy
project monies for 15 contractors serving 26 sites in Arizona. Contractors include 10 school
districts, 3 community- based organizations, 1 college, and 1 adult education provider. For
1997, there are 13 contractors serving 24 sites. ADE did not renew contracts for k oVal ley
Unified School District's and Wlllcois Council for Family Concern's sites for 1997. Contract
award amounts are presented in Table 3 ( see page 23).
Two Model and Training Resource Contractors- Mesa Unified School District's Family
Tree Project and Pima County Adult Education served as model and training programs.
Both models also serve as contractors for a total of 6 sites. The models provide technical
assistance and training resources for the 26 sites. Mesa Urufied School District serves 13 sites
in Arizona's northern region ( including Maricopa County), and Pima County Adult
Education serves the remaining 13 southern Arizona sites.
The number of sites each model program will support for school year 1996- 97 has been
reduced by one each because the sites operated by the Council for Family Concern and
Chmo Valley Urufied School District were closed for failure to fulfill their contracts. Their
grants were not renewed for the 1996- 97 school year.
Table 3
Arizona Family Literacy Pilot Program
Contractors and Contract Amounts
Years Ended or Ending June 30,1995,1996, and 1997
( Unaudited)
Contractor
Apache County
Red Mesa USD No. 27
Cochise County
Cochise Community College
Council for Farmly Concern
Coconino County
Flagstaff USD No. 1 ( 2 sites)
Maricopa County
Isaac ESD No. 5 ( 2 classes)
Literacy Volunteers of Maricopa County
Mesa USD No. 4 ( 3 sites)
Phoenix Indian Center
Tempe ESD No. 3
Navajo County
Pinon USD No. 4
Pima County
Pima County Adult Education ( 3 sites)
Tucson USD No. 1 ( 5 sites) '
Yavapai County
Chino Valley USD No. 51
Yuma County
Crane ESD No. 13 ( Morning Session)
Crane ESD No. 13 ( Afternoon Session)
Somerton ESD No. 11
Model Programs
Mesa USD No. 4
Pima County Adult Education
Total
Tucson USD No. 1 has four sites for 1997.
Source: Auditor General staff analysis of data provided by the Arizona Department of Education.
Propm service provideus given extensive opportunities fou family literacy training- Prior
to beginning the 1996- 97 school year, Arizona's Family Literacy Pilot Program staff received
training in implementing a family literacy program. Staff of the Adult Education Division
collaborated with the two model programs, and the Arizona Adult Literacy and Technology
Resource Center, Inc., to provide a week- long summer institute in August. The
comprehensive statewide training included Family Literacy Pilot Program staff, program
coordinators and administrators, and individuals from other family literacy programs, such
as Arizona's Even Start staff. The training was supported by federal adult education monies.
All sites were represented at the institute. About 200 individuals participated in the in- depth
5- day training, which focused on families' transfer of literacy experiences to the home.
Sessions detailed the implementation and integration of the four critical family literacy
components and other important components of family literacy, such as home visits. In
addition, participants took part in interactive activities and discussions to practice and
review training information.
The extensive statewide training was the fourth such activity sponsored by ADE for family
literacy staff since the Program's inception. Previous training sessions were conducted in
June, August, and December 1995, and January 1996.
3. lnformation on program revenues and expenditures.
Appropriation for the Family hteracy Pilot Program totaled $ 975,000 in school year 1994- 95
and $ 1,000,600 in school year 1995- 96. School years 1996- 97 and 1997- 98, respectively, have
each been appropriated $ 1,000,000. Family Literacy Pilot Program monies for 1994- 95 were
nonreverting, allowing ADE to carry over unspent monies.
ADE reduced 1995- 96 contractors' budgets by 12 percent to correct budgeting errors made
in school year 1994- 95. Sites were forced to revise their budgets mid- year to operate on the
reduced monies. Program budgets for 1996- 97 have been cut an additional 24.5 percent from
the 1995- 96 levels.
4. lnformation on the number and characteristics of enrollment and disenrollment.
The legslation and contracts require each site to enroll no fewer than 10 and no more than
15 famihes. Enrollment numbers varied at sites due to attrition and efforts to retain students.
Attendance records submitted by contractors showed 12 sites served an average of 10 or
more families for the 1995- 96 school year. The remaining sites were out of compliance with
the legislation.
Enrollment information showed fluctuating attendance rates for programs. The total number
of families enrolled at each site for the school year varied from 12 to 33. In addition, the
average number of families sites served throughout the school year ranged from 4 to 17.
Due to the poor quality of the data submitted by the contractors, it is not yet possible to
determine the causes of disenrollment
5. Information on the average cost for each family in the program.
Family literacy is based on the premise that a program incorporating adult, child, and parent
education is more effective than independent adult or early childhood components. The
complexity and variety of services provided by a family literacy program requires multiple
resources. In addition, budgets required to operate a family literacy program can vary
depending on the number of days per week the program operated, the program size, and
cost of conducting the program in a community, NCFL estimates a full- time program
requires a budget of $ 50,000 to $ 90,000 per year.
Cost per family- Family Literacy Pilot Programs spent an average of $ 5,198 of state monies
per family for family literacy services.
Office of the Auditor General staff calculated the cost per family using state monies allocated
to contractors for school year 1995- 96, and the average number of families served by
program contractors.
Costs va y substarztiully across progratrts- Cost per family varied by site for school year
1995- 96. Some variation is attributed to the different amount of monies allocated to each
site, the number of families served, and geographic locations of programs.
The cost per family ranged from $ 2,168 to $ 17,317. The most cost- effective site, whch was
Longfellow, in the Mesa Unified School District, also served the hghest average number
of farnhes for the school year. The site's state monies were $ 36,848, and the average number
of families served was 17. The number served exceeds the maximum number stipulated
by the Legislature. In contrast, the site with the least cost- efficient services, Pinon, served
an average of 7 families for the school year. This site also received the highest amount of
monies from ADE, $ 121,222. Pinon Unified School District operates in an isolated area with
few accessible resources. Table 4 ( see page 26), provides the costs per family for the family
literacy sites.
Table 4
Arizona Family Literacy Pilot Program
State Cost per Family per Site
Year Ended June 30,1996
( Unaudited)
Calculated Cost
per Family Based
Site State Monies on 15 Families
Actual Cost
per Family
Crane ESD No. 13
Crane Morning Session
Crane Afternoon Session
Flagstaff USD No. 1
Wip
L~ UPP
Isaac ESD No. 5
Isaac English for Speaker of Other Languages -
Isaac GED
Mesa USD No. 4
Eisenhower
Lincoln
Longfellow
Pima County Adult Education
Liberty
Nash
Ochoa
Tucson USD No. 1
Lawrence
MeyersGanoung
Pueblo Gardens
Tucson High School
Wakefield
Contractors with single classmrn sites
Cochise Community College
Literacy Volunteers of Maricopa County
Phoenix Indan Center
Pinon USD No. 4
Red Mesa USD No. 27
Somerton USD No. 11
Tempe ESD No. 3
Chino Valley USD No. 51 '
Council for Family Concern '
Total
Average
' Chino Valley USD No. 51 and Council for Family Concern in Willcox are included in the total state dollars, and their contract amounts are
included in the total per family cost calculation, but no families are attributable to them.
Source: Auditor General Staff analysis of data provided by the Arizona Department of Education F d y Literacy contractors.
Compamble propms' cost comparisons - Cost comparisons with other similar programs,
such as Arizona Even Start, show those programs to be providing family literacy services
at a lower cost per family than the Arizona Family Literacy Pilot Program sites. Arizona
Family Literacy Pilot Programs and Even Start both deliver family literacy services to
parents and their chddren. The differences between the two programs are their sources of
financial support, and their age eligibility requirements for cluldren. Even Start is a
federally supported program that provides services to families with children ages 0 to 7,
while the Arizona Family Lteracy Pilot Program serves famihes with three- and four- year-old
chddren. Additionally, Even Start has no class size limitation or citizenslup eligibility
requirement.
Arizona's eight federally supported Even Start programs closely resemble the Family
Literacy Pilot Program sites; in fact, there is collaboration between Family Literacy Pilot
Program and Even Start sites. Programs collaborate to reduce service costs and increase
available resources, and both programs adhere to NCFL models. In comparison to Arizona's
Family Literacy Pilot Program, Arizona Even Start sites have a lower cost per family.
Other comparable programs in the country, such as the federally supported Head Start
Family Service Centers, privately supported Kenan Family Literacy Project, and PACE,
provide similar services to families at lower costs. These programs are considered family
support programs, and provide similar educational and parenting services to families. Cost
comparisons show these programs to provide services at a lower cost, with the exception
of the Kenan Family Literacy Project, which has a range of costs that may fall below or
exceed Arizona Family Literacy Pilot Program's average cost. Table 5 lists these programs
and their corresponding cost per family:
Table 5
Arizona Family Literacy Pilot Program
Cost of Comparable Programs
Program
Arizona Family Literacy Pilot Program
Arizona Even Start '
Head Start Family Service Centers
Kenan Family Literacy Project
Annual Cost per Family
Even Start costs include local contributions.
Source: Auditor General staff analysis of data provided by the Arizona Department of Education, Family Literacy
Program contractors, and information published on other programs.
6. Information concerning progress of program participants in achieving goals and
objectives.
This report does not address the progress of participants in aclueving program goals and
objectives. Information measuring progress toward increased literacy, better parenting, and
developmental progress for preschool children will be available in 1997, after programs
deliver services for the second full program year. However, due to the difficulty the
contractors have enrolling and retaining participants in the Program, the outcomes
evaluation will be limited by the small numbers of families who have been in the Program
long enough to have been assessed on the outcomes measures at least two times.
7. Recommendations regarding program administration.
In Finding I ( see pages 5 through ll), it is recommended that:
Family Literacy Pilot Programs should address the following recommendations to correct
for noncompliance and improve program services.
Implement and integrate the four family literacy components consistently
Increase retention efforts to maintain 10 to 15 families at all times
Collaborate with other organizations
Conduct regular home visits that are consistent and purposeful.
In Finding II ( see pages 13 through 15), we recommend that:
ADE should actively monitor sites for compliance with requirements and service quality.
Family literacy administrators should collaborate with model programs and other ADE
divisions to develop a monitoring plan and conduct site visits. Collaboration with
individuals from Even Start and early chddhood education would improve comprehensive
oversight of family literacy components.
ADE's Adult Education Division should collaborate with the model programs to determine
their capacity to assist with the program monitoring process.
In Finding III ( see pages 17 through 19), we recommend that:
ADE should provide monies for fewer sites for school year 1998 to ensure that all the sites
can operate at a level of quality that has the potential to produce the desired family learning
outcomes.
8. Recommendations for program expansion.
The Office of the Auditor General cannot address recommendations pertaining to program
expansion until second- year outcome data is available. However, even after tkus data is
available, the small numbers of families who have received family literacy services for a
sigruficant and sustained length of time may lessen the power of the outcome evaluation,
Consequently, there may be h i t s to the extent to which generalizations about the Program
can be drawn and recommendations made.
( This Page Intentionally Left Blank)
Agency Response
( This Page Intentionally Left Blank)
/ Lisa Graham Keegan
Sunerintendent of
State of Arizona
Department of Education
public Instruction
December 23, 1996
Mr. Douglas R. Norton
Auditor General
29 10 North 44th Street, Ste. 4 10
Phoenix, AZ 8501 8
The Family Literacy Pilot project completed the first full year of operation on June 30,
1996. During the year your agency has worked closely with Department of Education
staff in conducting monitoring activities and on- site visitations.
Your report reflects some of the results of those evaluation exercises. Thank you for
sharing the preliminary report draft with my office. We share your concerns for program
improvement and agree with several of the reports recommendations.
The key words and understanding our agencies must embrace at this time is that the
" pilot" is a program under development. Our concern and issues must be framed in the
context of programs on a " pilot" basis moving toward, but having not yet reached, our
desired program design and ultimate goal for an intergenerational educational initiative.
Your report will be of assistance to our continued efforts to provide children and adults
with access to extraordinary education.
Please be assured that our contracts to local providers have not, nor will they exceed, the
funds appropriated by the Arizona State Legislature.
Our meeting with Debra Davenport and your staff on December 13"' was productive. As
the " pilot" continues to improve we will be able to measure progress toward increased
literacy, better parenting and developmental progression for preschool children. The
second full program year will reveal program output measures.
We concur with your reports statement, " Overall, sites are serving structurally intact
families who meet the eligibility criteria and aspire to improve their literacy skills and
reduce poverty." This statement is supported by your data, if you show the reverse of
Table 1 on page 7, which indicates eighty percent of the pilot sites are meeting the
legislated program goals.
1535 West Jefferson, Phoenix, Arizona 85007 602- 542- 4361
During the second full year of the " pilot" our agency in concert with your office will
increase project evaluations, on- site visitations and staff development by the two model
sites. We will not increase state staff positions to accomplish this. Our agency is
assigning current staff to complete our technical assistance responsibilities.
The Arizona family literacy pilot program is a relatively new innovation in education.
Many states are modeling our efforts to provide communities with the incentive and
opportunity to evolve new programs from existing proven programs such as Family
Literacy.
Our working together will help us document promising results in programs that are
intensive, long term, and pay attention to quality in all components. Lessons are also
being learned that let us know we are on the right track by creating a synergy through
implementing our four components of family literacy. Value is added to our effort when
learning occurs not only across generations but across disciplines as Arizona adult and
child educators experience reciprocal learning and teaching.
Our agency looks forward to our continued work together and the submission of positive
reports to those you are charged to inform.
Sincerely, + )& AJL2bd+ 8J
Lisa Graham Keegan b~
Superintendent of Public Instruction
Appendix
( This Page Intentionally Left Blank)
Appendix A
Family Literacy Data
Collection Methodology
The Arizona Famdy Literacy Pilot Program evaluation process used various instruments and
methods for data collecting. Two instruments were used to collect data during site visits, the
site observation rating form and the adult and early childhood education interview rating form.
An additional form, the contractor survey, was used to collect logistical data from contractors.
Other information collected from contractors consists of daily class schedules and monthly
attendance records.
Site Observation Rating Form
This form is a condensed version of the National Family Literacy Program Rating Scale. It is
used for rating family literacy sites based on guidehes withm the 4 family literacy components,
and consists of 52 statements about model family literacy practices. The rating for each
statement ranges from not assessed ( 0) to fully implelrrented ( 3).
Adult and Early Childhood
Education Interview Rating Form
This form is used to interview family literacy program staff. Staff qualifications and opinions
about program progress are recorded on this form. It contains 40 questions consisting of both
closed- and open- ended questions. Twenty- nine of the questions are adapted from the National
Family Literacy Program Rating Scale. These questions are rated as NA ( not assessed), Y ( Yes),
and N ( No) according to the response.
Contractor Survey
The Contractor Survey is a self- reporting instrument that reports information about program
logstics. It consists of 21 questions about contractor procedures, services, qualifications, and
funding.
a- i
