Performance audit, Arizona Department of Education, information management

              Performance Audit
Arizona Department
of Education–
Information Management
Performance Audit Division
Debra K. Davenport
Auditor General
AUGUST • 2006
REPORT NO. 06 – 07
A REPORT
TO THE
ARIZONA LEGISLATURE
The is appointed by the Joint Legislative Audit Committee, a bipartisan committee composed of five senators
and five representatives. Her mission is to provide independent and impartial information and specific recommendations to
improve the operations of state and local government entities. To this end, she provides financial audits and accounting services
to the State and political subdivisions, investigates possible misuse of public monies, and conducts performance audits of
school districts, state agencies, and the programs they administer.
The Joint Legislative Audit Committee
Representative Laura Knaperek, Chair Senator Robert Blendu, Vice Chair
Representative Tom Boone Senator Ed Ableser
Representative Ted Downing Senator Carolyn Allen
Representative Pete Rios Senator John Huppenthal
Representative Steve Yarbrough Senator Richard Miranda
Representative Jim Weiers (ex-officio) Senator Ken Bennett (ex-officio)
Audit Staff
Melanie M. Chesney, Director
Joseph D. Moore, ITS Director and Contact Person
Dot Reinhard, Manager
Estella Arredondo Emily Chipman
Allan Friedman Melynda Harris
Jenner Holden Thomas Huber
Copies of the Auditor General’s reports are free.
You may request them by contacting us at:
Office of the Auditor General
2910 N. 44th Street, Suite 410 • Phoenix, AZ 85018 • (602) 553-0333
Additionally, many of our reports can be found in electronic format at:
www.azauditor.gov
DEBRA K. DAVENPORT, CPA
AUDITOR GENERAL
STATE OF ARIZONA
OFFICE OF THE
AUDITOR GENERAL
WILLIAM THOMSON
DEPUTY AUDITOR GENERAL
2910 NORTH 44th STREET • SUITE 410 • PHOENIX, ARIZONA 85018 • (602) 553-0333 • FAX (602) 553-0051
August 17, 2006
Members of the Arizona Legislature
The Honorable Janet Napolitano, Governor
Mr. Tom Horne, State Superintendent of Public Instruction
Arizona Department of Education
Transmitted herewith is a report of the Auditor General, A Performance Audit of the Arizona
Department of Education—Information Management. This report is in response to Arizona
Revised Statutes (A.R.S.) §41-2958 and was conducted under the authority vested in the
Auditor General by A.R.S. §41-1279.03. I am also transmitting with this report a copy of the
Report Highlights for this audit to provide a quick summary for your convenience.
As outlined in its response, the Department of Education agrees with all but one of the
findings and plans to implement all but one of the recommendations.
My staff and I will be pleased to discuss or clarify items in the report.
This report will be released to the public on August 18, 2006.
Sincerely,
Debbie Davenport
Auditor General
Enclosure
The Office of the Auditor General has conducted a performance audit of the Arizona
Department of Education’s (ADE) information management function pursuant to
Arizona Revised Statutes (A.R.S.) §41-2958. This audit was conducted under the
authority vested in the Auditor General by A.R.S. §41-1279.03 and is the third in a
series of three reports regarding ADE. The other two audit reports address aspects
of ADE’s administration and allocation of funds and its accountability programs.
Background
ADE depends on information technology (IT) to manage its own operations and to
provide state-wide oversight and assistance to Arizona’s 238 school districts and 487
charter schools. IT is important to ADE’s processes for calculating and administering
the transfer of approximately $3 billion in state appropriations to districts and charter
schools. It is also important for state-wide reporting on education initiatives such as
AZ LEARNS and No Child Left Behind (NCLB). Many different groups rely on ADE’s
IT resources, including parents, teachers, schools, and the Legislature for various
purposes such as obtaining tutoring information, applying for teaching certificates
and positions, or obtaining information about Arizona’s schools, their students, and
their finances. At the heart of ADE’s technology efforts is its Student Accountability
Information System (SAIS). SAIS was established by statute to enable school districts
and charter schools to transmit student-level data and school finance data
electronically through the Internet to ADE to comply with ADE’s and the State Board
of Education’s statutory obligations.
ADE needs to better manage security of its information
technology systems and operations (see pages 9
through 16)
Sensitive information, such as social security numbers, has been exposed because
of security weaknesses in ADE’s Web-based applications. However, auditors found
no indication that the security weaknesses identified have yet been exploited or that
sensitive information has been compromised.
Office of the Auditor General
SUMMARY
page i
The Auditor General hired an independent consulting firm to help its auditors assess
12 of ADE’s 76 Web-based applications. These 12 applications are among the most
frequently used and/or contain the most sensitive data. The auditors found 8 different
types of critical vulnerabilities. Most of the applications had 5 or more of these
vulnerabilities.
These weaknesses could potentially:
􀁺􇩅 Enable an attacker to obtain access to other users’ accounts, view sensitive
information, or perform functions they are not authorized to; or
􀁺􇩐 Provide attackers with useful information that could assist them in carrying out
further attacks.
In response to the Web-based application vulnerabilities found in this audit, ADE
began to take steps to identify critical applications and prioritize actions needed to
assess and correct discovered vulnerabilities. ADE management asserts those
activities were performed during May and June 2006.
However, the vulnerabilities in ADE’s applications are only part of the overall security
weaknesses in ADE’s IT operations. While ADE has policies and procedures in place
for many important security objectives, auditors found that for 12 of 13 areas
reviewed, these policies and procedures were either inadequate or ineffective,
including those over such functions as:
􀁺􇩐 Password management;
􀁺􇩉 Installing security updates for their IT systems; and
􀁺􇩍 Monitoring systems to detect attacks and/or identify unauthorized uses.
Although previous security assessments have alerted ADE to many security
concerns, it has not always taken sufficient action to address them. For example,
while ADE did fix some specific Web-based application weaknesses found in a 2004
security assessment of its systems, other weaknesses noted in the reports were not
addressed. Several of these weaknesses were serious in nature and should have
been addressed immediately.
ADE has indicated that it has not had a coordinated approach for addressing
security concerns that it identifies or that come to its attention. ADE should consider
creating an appropriate position to be responsible for all IT security within ADE. The
reporting line of the security position should be such that it can effectively design,
implement, and enforce compliance with the organization’s security policies,
standards, and procedures, and ensure that they are functioning effectively.
State of Arizona
page ii
ADE can further enhance SAIS' reliability (see pages 17
through 28)
As the primary entity responsible for operating and maintaining SAIS, ADE plays an
important role in helping ensure SAIS’ reliability. SAIS was established by statute to
enable school districts and charter schools to electronically transmit student-level
data and school finance data to ADE. According to a survey conducted by this
Office, although the majority of SAIS users have confidence in the accuracy of SAIS
data, some users continue to have concerns. Approximately 29 percent of SAIS
users who auditors surveyed reported that they were not confident that the data in
SAIS was accurate when it was needed for final reporting or funding purposes, and
in 2005 the Arizona School Administrators adopted a formal position that data within
SAIS is unreliable. ADE can further improve the reliability and processing
performance of SAIS by (1) adding more controls such as data variance checks to
ensure reliability; (2) taking steps to improve functionality such as providing the
capability to archive reports; and (3) monitoring the performance of and potentially
rating the various software packages that districts and schools use to submit data to
SAIS.
ADE has established some data validation controls, and improving these controls
would help further ensure data accuracy. Input controls are designed to help ensure
that the data entered into SAIS is accurate. Although some input controls are in
place, ADE should add another type, known as variance checks. Variance checks
would alert ADE to unusual changes in the data such as a large change in the
number of students from the prior year. ADE also needs to add automated process
controls to SAIS. Process controls are designed to ensure that no data is added, lost,
or altered during processing. Adding some process controls, such as data
reconciliation at key points in the processing of SAIS data, would also help enhance
data accuracy.
ADE can also take some basic steps to improve SAIS’ functioning. For example, ADE
should continue its efforts to establish the ability to archive SAIS reports so that users
will have access to historical report data. ADE should also establish a tactical team
comprising internal and external stakeholders to identify and prioritize other changes
to SAIS that can address users’ concerns.
Finally, ADE also needs to improve its oversight of the software packages that SAIS
users use to transmit data to SAIS. School districts and charter schools develop or
buy student management system (SMS) software packages to electronically submit
data to SAIS. There are at least 20 different packages in use. Auditors found that the
quality of these packages varies considerably. Based on information provided by
ADE, one software package had errors in only 0.7 percent of the data submitted,
while another package had a 31.8 percent error rate. There are several reasons the
software quality varies, including ADE’s relaxation of testing requirements. According
Office of the Auditor General
page iii
to ADE, resource limitations have prevented it from continuing its original vendor
testing requirements. However, to further enhance SAIS data reliability, ADE should
take action to enhance SMS software performance by monitoring it and should
consider rating or certifying SMS software packages based on performance.
ADE needs to improve IT project management and
operations oversight (see pages 29 through 34)
ADE needs to improve its approach for managing systems development projects
and for overseeing its IT operations. The ADE IT section, which currently has 57
positions, is primarily responsible for maintaining all of ADE’s information technology
resources and for managing ADE’s internal and external networks. There is a lack of
consistency in how systems are developed—both by the IT section and by other
units within ADE—and many systems have not been developed following a standard
agency-wide development process. Many of ADE’s information systems lack good
documentation, which has caused difficulties in maintaining and modifying the
systems. ADE needs to ensure that systems development projects follow a standard
approach and are well documented. ADE also needs to ensure that this standard
approach applies to all of ADE’s projects, whether developed by the IT section or by
other ADE units.
The IT section should also improve key IT oversight operations. It should develop
performance indicators, conduct risk assessments of threats to its operations, and
prepare an adequate business continuity plan.
ADE needs to ensure its information technology meets its
business needs (see pages 35 through 39)
Although IT is key to many of ADE’s administrative functions and responsibilities,
ADE has not adequately ensured that its IT resources meet its business needs.
Specifically, IT management or staff do not adequately participate in developing
ADE’s strategic plan, despite the fact that IT resources may be needed to ensure that
objectives are met. ADE also lacks a department-wide process for prioritizing IT
projects. ADE needs to take at least two steps to ensure that IT can effectively
support its business goals and objectives, including:
􀁺􇨠 Establishing an IT steering committee. The committee should include
representatives from senior management, user management, and the IT
function. Once established, the committee should be responsible for things
such as ensuring adequate IT involvement in the department and division
State of Arizona
page iv
planning processes, providing overall IT direction, and ensuring that adequate
processes exist for identifying, funding, and allocating department-wide IT
resource costs.
􀁺􇩃 Considering whether the IT section’s placement within the organization
adequately ensures that IT can effectively meet ADE’s business goals and
objectives.
In addition, the IT section’s internal planning process is limited and does not identify
the resources it needs to maintain and operate its current systems or plan for future
IT initiatives. Therefore, ADE should ensure that the IT section establishes an effective
planning process. This process should include developing a strategic plan with input
from key stakeholders that defines long-term direction, and developing an action
plan that, among other things, formulates strategies; evaluates costs, benefits, and
possible consequences of alternative courses of action; assigns responsibility for
implementation; and determines the resources necessary to carry out the plan.
ADE not in full compliance with student-level data
collection notification and disposal requirements (see
pages 41 through 42)
ADE is not complying with two data-related statutory requirements. ADE has not
complied with a requirement that it tell school districts and charter schools the
specific statutory authority for each item of student-related data that it requires them
to submit. However, ADE has begun to compile this information and hopes to publish
it by the middle of fiscal year 2007.
ADE has also not complied with a requirement that it adopt guidelines to remove
outdated student information from SAIS. ADE explains that it has not adopted
guidelines due to state and federal initiatives that suggest ADE may need to retain
education data for longer periods than statute currently calls for. However, it appears
that statute does not prescribe a retention period for student-level data. Therefore,
ADE should adopt a retention schedule for such data and adopt guidelines to
remove the outdated data in SAIS in accordance with state statute.
Office of the Auditor General
page v
State of Arizona
page vi
Office of the Auditor General
TABLE OF CONTENTS
continued
1
9
9
11
15
16
17
17
18
19
23
28
29
29
31
33
34
Introduction & Background
Finding 1: ADE needs to better manage security of its
information technology systems and operations
Sensitive information exposed and system vulnerabilities exist
ADE has not adequately addressed important security concerns
ADE should consider assigning responsibility for IT security
Recommendations
Finding 2: ADE can further enhance SAIS’ reliability
SAIS processes and maintains a large amount of valuable data
Concerns regarding SAIS
ADE should take additional steps to improve SAIS data reliability
and processing
ADE should take steps to improve oversight of Student
Management System (SMS) software performance
Recommendations
Finding 3: ADE needs to improve IT project
management and operation oversight
IT project management approach lacking
ADE should establish effective process for developing information
systems
ADE should improve key oversight activities
Recommendations
page vii
State of Arizona
TABLE OF CONTENTS
Finding 4: ADE needs to ensure its information
technology meets its business needs
Coordination of IT efforts limited
IT section’s planning process ineffective
Recommendations
Finding 5: ADE not in full compliance with student-level
data collection notification and disposal requirements
Recommendations
Glossary
Agency Response
Table:
1 Analysis of IT Security Objectives at ADE
35
35
37
39
41
42
a-i
13
concluded
page viii
The Office of the Auditor General has conducted a performance audit of the Arizona
Department of Education’s (ADE) information management function pursuant to
Arizona Revised Statutes (A.R.S.) §41-2958. This audit was conducted under the
authority vested in the Auditor General by A.R.S. §41-1279.03 and is the third in a
series of three reports regarding ADE. The other two audit reports address aspects
of ADE’s administration and allocation of funds and its accountability programs.
ADE is highly dependent on information technology
ADE is highly dependent on the use of information technology (IT). IT is critical to
ADE’s ability to manage its own operations and to provide state-wide oversight and
assistance to Arizona’s 238 school districts and 487 charter schools. For example, IT
is important to ADE’s processes for calculating and administering the transfer of
approximately $3 billion in state appropriations to districts and charter schools, and
for state-wide reporting on education initiatives and mandates such as AZ LEARNS,,
Arizona’s Instrument to Measure Standards (AIMS), and the federal No Child Left
Behind (NCLB) requirements.1— 3
ADE operates its own internal computer network and has developed and maintains
a number of computer systems to meet its business needs. These systems are used
to perform a variety of functions, including the management and tracking of the
expenditure of state and federal funds for the programs it administers, and the
collection and reporting of information about students, teachers, and schools in
Arizona. Many of these systems are accessible over the Internet.
Many different groups rely on ADE’s IT resources. ADE staff relies on its systems to
carry out many of their day-to-day responsibilities. Members of the public, including
IT is critical to ADE’s
own operations and
oversight of and
assistance to Arizona’s
public education
programs.
Office of the Auditor General
INTRODUCTION
& BACKGROUND
page 1
1 AZ LEARNS is ADE’s school accountability system that it also uses to meet NCLB’s accountability requirements.
2 AIMS is a standards-based test given to students in Arizona. Effective for the graduating class of 2006, high school
students are required to pass AIMS tests in order to graduate. AIMS is used to help measure students’ progress toward
mastering reading, writing, and mathematics standards that the Arizona Board of Education established.
3 NCLB was signed into law in January 2002. Under the act’s accountability provisions, states must describe how they will
close the achievement gap between students and make sure all students, including those who are disadvantaged,
achieve academic proficiency. States must produce annual report cards to inform parents and communities about state
and school progress.
parents and students, may use ADE’s Web site to obtain school information, such as
school report cards, and AIMS information, such as requirements, study guides,
tutoring information, and school or district test results. Teachers may use the Web site
to obtain information about educational standards, curriculum development
resources, and study guides for AIMS tests, and to apply for teaching certificates or
teaching positions throughout the State. Districts, schools, and charter schools use
ADE’s Student Accountability Information System (SAIS) to submit electronic
information, such as district and school identification codes, and students’ names,
birthdates, identification numbers, absences, and special education enrollment
information to ADE. Policymakers, such as the Legislature, also rely on ADE’s ability
to provide information from their systems about Arizona’s schools, their students,
and their finances.
Because so many individuals and groups rely on ADE’s IT resources, and because
of the quantity and type of information that ADE maintains on its systems, it is
especially important that ADE’s systems be secure and that they produce accurate
and complete information in a timely manner. Security is particularly important
because of the sensitive nature of some of the information that ADE maintains on its
systems, such as students’ names, birthdates, and attendance records, and
teachers’ names and social security numbers. Additionally, given ADE’s dependence
on IT, it must ensure that its IT resources are developed and maintained
appropriately, operated effectively, available when needed, and meet its business
needs.
SAIS a key component of ADE’s IT operations
SAIS is a key component of ADE’s technology efforts. SAIS was established by
statute to enable school districts and charter schools to transmit student-level data
and school finance data electronically through the Internet to ADE for the purposes
of complying with the statutory obligations of ADE and the State Board of Education.
Districts and charter schools are required by statute to submit electronic data on a
school-by-school basis, including student-level data, to ADE so that they can receive
monies for the cost of educating students. Districts and charter schools were
originally required to begin submitting this data to ADE by July 1, 2001. However,
according to agency officials, the system did not become fully functional until the
2002-2003 school year. SAIS is used as the basis for calculating Average Daily
Membership (ADM) and distributing about $3 billion in state appropriations to
districts and charter schools.1 Funding for school districts is based on prior year
student attendance data with some adjustments made for certain exceptions, such
as high-growth school districts. Funding for charter schools is based on current year
attendance data.
ADE’s IT systems need
to be secure and
produce accurate and
complete information in
a timely manner.
1 ADM is defined in A.R.S. §15-901 and is the total enrollment of fractional (i.e. less than full-time) and full-time
students, minus withdrawals, of each school day through the first 100 days or 200 days in session, as applicable, for
the current year.
State of Arizona
page 2
SAIS’ operation involves significant electronic data processing as well as manual
effort for both SAIS users and at ADE. Information entered into SAIS is initially
recorded in individual SAIS users’ data systems, referred to as Student Management
Systems (SMS). According to agency information, at least 20 different SMS software
packages, developed either by districts or purchased from outside vendors, are used
to collect and transmit student data to SAIS. SAIS users are not required to submit
information to ADE more often than once every 20 school days. The information must
be sent in specific formats, which ADE defines. SAIS users upload their information
on the Internet to ADE, where it is then added into the SAIS databases. SAIS receives,
validates, and processes the data required by statute to be submitted by Arizona’s
districts and charter schools. In fiscal year 2006, approximately 46.5 million student
data transactions were submitted, such as records of student attendance or absence
from school, addition of new students, changes in student schools, etc.
Between fiscal years 1997 and 2006, ADE reports spending approximately $11
million for costs associated with SAIS' development and maintenance. In addition,
ADE provided about 90 percent of the $4.5 million it received through Proposition 301
in fiscal year 2002 to districts and charter schools to help them develop system
interfaces.
IT section
ADE has an IT section that reports to the Associate Superintendent for Education
Policy. This section is primarily responsible for maintaining all of ADE’s information
technology resources and for managing ADE’s internal and external networks.
However, some IT functions, including some system development efforts, occur
within other ADE units, such as the School Finance unit.
As of June 2006, ADE’s IT section reports that it had 57 FTEs with 7 vacancies in 9
functional groups, as follows:
􀁺􇩄 Development (23 positions, 5 vacancies)—This group is responsible for the IT
systems’ analysis, design, testing, implementation, and maintenance.
􀁺􇩒 Requirements, Analysis, and Testing (7 positions, 0 vacancies)—This group is
responsible for analyzing business needs, translating those needs into IT
requirements, developing testing scenarios, and conducting software testing.
􀁺􇩐 Program Management (6 positions, 2 vacancies)—This group is responsible for
feasibility studies related to software solutions, coordinating estimation teams,
developing project plans, providing customer liaisons, assisting in prioritizing
projects and maintenance efforts, and managing projects.
In fiscal year 2006,
SAIS processed
approximately 46.5
million student data
transactions.
Office of the Auditor General
page 3
􀁺􇩎 Network Services (4 positions, 0 vacancies)—This group is responsible for
administering and maintaining ADE’s computer network and communications
services.
􀁺􇩄 Database Administration (2 positions, 0 vacancies)—This group is responsible
for the design and creation of new databases, the maintenance and security of
existing databases, the monitoring of ADE’s file storage capacity, and the
backup and recovery of data maintained on ADE’s network.
􀁺􇩄 Desktop Support (4 positions, 0 vacancies)—This group provides computer
hardware and software support for ADE’s five locations in Phoenix, Flagstaff,
and Tucson; user training; equipment and software inventory control; and audio-visual
support; and develops IT purchasing standards and ensures compliance
with those standards when IT is purchased.
􀁺􇩐 Phone Support (3 positions, 0 vacancies)—This group provides technical
support for both internal and external users of Web-based IT systems, and for
internal users on agency-wide applications.
􀁺􇩗 Web Site Maintenance (1 position, 0 vacancies)—This individual is responsible
for maintaining the content of ADE’s Web site for all agency units, tracking and
reporting on Web site usage, and assisting with Web-based presentations.
􀁺􇩁 Administration (7 positions, 0 vacancies)—This group consists of the chief
information officer, an administrative assistant, and the directors of the units
above.
In fiscal year 2006, according to ADE’s Business and Finance unit, the total IT section
budget was approximately $5.7 million. The IT section’s budget is partially funded by
other divisions within ADE through a cost-recovery model.1 The IT section’s $5.7
million budget also consists of Proposition 301 monies for SAIS maintenance, and
federal funds for program development related to federal requirements. In addition to
the IT section, some divisions within ADE also have small IT staffs and expend
monies on IT-related activities that are separate and apart from the IT section’s
expenditures.
Scope and methodology
This audit focused on information management within ADE and covered five areas:
security over ADE’s IT systems and resources, SAIS data reliability, IT project
management and operations oversight, the IT section’s ability to meet ADE’s
business requirements, and ADE’s compliance with student-level data collection
1 In fiscal year 2006, this included approximately $500,000 in current-year funding and $200,000 in carryover from previous
fiscal years.
State of Arizona
page 4
notification and disposal requirements. This report presents five findings and
associated recommendations, as follows:
􀁺􇩁 ADE needs to improve its management of information technology (IT) security.
Although auditors found no evidence that sensitive information has been
exposed, auditors found a lack of adequate security over ADE’s Web-based
applications. ADE needs to address important security concerns and should
consider creating a position with specific responsibility for IT security;
􀁺􇩁 ADE can take additional steps to improve SAIS data accuracy and functionality.
Although ADE has some data validation controls, enhancing those controls
would help further ensure data accuracy. In addition, providing the ability to
archive SAIS reports would allow users access to historical data. ADE could also
take steps to help improve SMS software packages by monitoring performance
and should consider rating or certifying these packages;
􀁺􇩁 ADE needs to improve management of the IT systems it develops and of its IT
operations overall. A more structured process for developing and maintaining its
IT systems would help ensure that systems are developed appropriately and
that they can be maintained effectively. Performing activities such as regularly
monitoring key performance measures, conducting risk assessments, and
maintaining and testing a business continuity plan would help ADE ensure
effective IT operations;
􀁺􇩁 ADE needs to better coordinate its IT efforts in order to ensure that IT is capable
of meeting ADE’s business requirements, and that it properly aligns with ADE’s
goals and priorities. ADE’s IT section lacks an effective strategic planning
process, and steps should be taken to ensure that it adequately plans. ADE
should consider the placement of IT within the organization to ensure that IT
requirements are appropriately addressed; and
􀁺􇩁 ADE has not fully complied with statutory requirements for student-level data
collection notification or information disposal guidelines. ADE should comply
with reporting requirements by adding, to its Web site listing of SAIS data
elements, a reference to the statutory authority necessitating the collection of
each SAIS data element. ADE should also develop guidelines for removing
outdated data from SAIS.
Several methods were used to study the issues addressed in this audit. Methods
used in all areas included interviews with ADE management and staff and other
stakeholders, including district, school, and charter school staff. Auditors also
reviewed Arizona Revised Statutes, and several ADE documents such as policies
and procedures, strategic plans, and budget documents. A number of best practice
Office of the Auditor General
page 5
guides were also reviewed.1 In addition, the following methods were used in
reviewing each specific area:
􀁺􇩔 To evaluate the security of ADE Web-based applications, auditors and an
independent security consultant retained by the Office of the Auditor General
tested the Web-based applications using both automated and manual security
testing techniques. To determine the status and effectiveness of ADE IT security
controls, auditors reviewed ADE IT security-related policies and procedures
and interviewed ADE staff. Auditors also performed manual testing of ADE IT
security procedures to determine effectiveness. To determine actions taken to
correct known security weaknesses, auditors reviewed past security
assessment reports, interviewed appropriate ADE staff, and performed work to
determine whether corrective actions were taken. Auditors also reviewed IT
security standards published by GITA, industry best practices, and federal IT
security guidelines.
􀁺􇩔 To evaluate the reliability and processing performance of SAIS, auditors
interviewed representatives from GITA, the Arizona School Administrators, and
several vendors of Student Management System (SMS) software products.
Additionally, auditors obtained and reviewed ADE documentation on SAIS,
including system documents, user manuals, and problem tickets, and reviewed
professional literature. To determine how other states maintain data reliability
and processing performance in their data management systems, auditors
interviewed other states’ education agencies.2 Auditors also conducted a state-wide,
Web-based survey of individuals who enter or review student data that is
submitted to SAIS. The purpose of the survey was to gather information about
the process of working with and correcting SAIS data errors; the quality of ADE
documentation, training, and support; and satisfaction with SMS vendors. The
survey was conducted between January 12, 2006 and January 30, 2006, and
was e-mailed to business officials who work for school districts, charter schools,
and charter holders. A total of 774 individuals received the e-mails. However,
auditors requested that e-mail recipients forward the survey on to any staff in
their district or charter who enter or review SAIS data. In total, auditors received
1 Best practice materials reviewed included: (1) Gallegos, Frederick, Sandra Senft, Daniel P. Manson, and Carol
Gonzales. Information Technology Control and Audit, 2nd Edition. Boca Raton, FL: CRC Press LLC, 2004; (2) Grance,
Tim, Joan Hash, and Marc Stevens. Security Considerations in the Information System Development Life Cycle. U.S.
Department of Commerce. Technology Administration. National Institute of Standards and Technology.Special
Publication. 800-64, Rev. 1. Gaithersburg, MD: NIST, June 2004; (3) IT Governance Institute. COBIT 4.0: Control
Objectives, Management Guidelines, Maturity Models. Rolling Meadows, IL: IT Governance Institute, 2005; (4) Ross,
Ron, Stu Katzke, Arnold Johnson, Marianne Swanson, Gary Stoneburner, George Rodgers, and Annabelle Lee.
Recommended Security Controls for Federal Information Systems. U.S. Department of Commerce. Technology
Administration. National Institute of Standards and Technology. Special Publication 800-53. Gaithersburg, MD: NIST,
Feb. 2005; (5) Office of Strategic Planning and Budgeting. Strategic Planning and Performance Measurement
Handbook, 3rd revision. Phoenix: OSPB, Jan. 1988.
2 Auditors interviewed staff at the California, Delaware, Florida, Massachusetts, Michigan, Ohio, Texas, and Wyoming
Departments of Education. These states were selected because they represented educational systems similar to
Arizona’s, had an IT system in use for more than 1 year, assigned unique IDs to students, were using or moving
toward use of data warehouse for storage, provided other agencies with the ability to look up student information,
and were recommended by other interviewees.
State of Arizona
page 6
338 responses. The majority of the survey responses came from those required
by statute to submit student data to SAIS. Specifically, 158 (47 percent) of all
survey responses were from district offices, and 95 (28 percent) were from
charter schools. In addition, auditors received survey responses from 52 (15
percent) district schools, 31 (9 percent) charter holders, and 2 (1 percent) third-party
groups (a technology consortium and an independent company) who
provide SAIS technical support to charter schools and districts.
􀁺􇩔 To determine if ADE provides adequate management and oversight of its
information technology systems development processes and its IT operations,
auditors observed IT systems and processes; reviewed ADE’s project
methodologies, procedures and policies, and system documentation; and
reviewed and applied information technology management best practice
information.
􀁺􇩔 To determine if ADE has established processes to ensure that IT is aligned with
its business needs and whether the IT section has developed a sufficient
strategic planning process to provide guidance and direction to its efforts,
auditors reviewed ADE planning and budget documents, and reviewed and
applied information technology management best practice information.
􀁺􇩔 To determine whether ADE is in compliance with student-level data collection
notification and disposal requirements, auditors reviewed applicable statutes
and consulted with legal counsel.
􀁺􇩔 To develop the Introduction and Background section, auditors compiled
information from state laws, interviews with agency officials and staff, unaudited
information from ADE’s Web site, federal government Web sites, and agency-prepared
documents such as organization charts, budget, and staffing
information.
Due to the technical nature of certain terms used in this report, a glossary of IT-related
terms is presented.
This audit was performed in accordance with government auditing standards.
The Auditor General and staff express appreciation to the Superintendent of Public
Instruction and the Arizona Department of Education and its staff for their
cooperation and assistance throughout the audit.
Office of the Auditor General
page 7
State of Arizona
page 8
ADE needs to better manage security of its information
technology systems and operations
ADE needs to improve its management of information technology (IT) security. Lack
of adequate security over its Web-based applications has resulted in exposure of
sensitive information and has left these applications susceptible to many critical and
well-known vulnerabilities. In addition, important IT security policies and procedures
are either missing or are ineffective, and ADE has not taken sufficient action to
address previously known IT security problems. To ensure that ADE adequately
protects its information and technology resources, it needs to take a number of
actions and should consider creating a position with specific responsibility for IT
security.
Sensitive information exposed and system vulnerabilities
exist
Sensitive information has been exposed due to security weaknesses in ADE Web-based
applications, which contain critical flaws, also referred to as vulnerabilities, that
could put them at further risk. Although auditors found no indication that these
security weaknesses have been exploited or that sensitive information has actually
been compromised, the mere existence of these weaknesses makes either possible.
ADE makes extensive use of Web-based applications accessible both over the
public Internet and through its internal network for use by its own staff. These
applications are used primarily to collect and report on information relating to
students, teachers, and schools in Arizona. A review of security over several of the
most frequently used applications revealed that some sensitive information could be
viewed by individuals who have no right or need to access it. In addition, the review
found that ADE applications were susceptible to a number of critical and well-known
vulnerabilities, which could potentially lead to further compromise of systems and
Office of the Auditor General
page 9
FINDING 1
State of Arizona
page 10
information maintained by ADE. In response to these findings, ADE has taken some
steps to address weaknesses in its Web-based applications.
ADE Web-based applications used extensively—ADE has approximately
76 active Web-based applications, such as those related to teacher certification,
child nutrition, grants management, and SAIS. These applications are used for a
variety of purposes, such as to collect and report on information about students,
teachers, and schools in Arizona. For example, student data, such as student
names, birthdates, and information about when students attend or are absent
from school, is submitted by school districts and charter schools through Web-based
applications. Information on certified teachers in the State, such as
teachers’ names, birthdates, and social security numbers, is also collected and
is accessible through these applications. Most of ADE’s Web-based applications
are available over the Internet to external users, such as school district, school,
and charter school officials, and to internal users such as employees at ADE.
ADE reports that approximately 3,300 different organizations and more than
13,600 individual user accounts have access to at least one of these applications.
Sensitive information exposed—A review of security over several of the most
often-used applications revealed that information, including some of a very
sensitive nature, was exposed and could be viewed by individuals who have no
right or need to access it. However, auditors found no indication that such
information maintained by ADE has been compromised.
The Office of the Auditor General contracted with an independent consulting firm
to assist with an assessment of 12 Web-based applications maintained by ADE.
The Web-based applications examined included those that are the most
frequently used and that contain and process the most sensitive information. For
example, applications that allow users to search for teacher certification
information, and to submit and view student data, were reviewed. The results of
this review revealed multiple circumstances that would allow a hacker or other
unauthorized individuals to obtain access to personally identifiable information,
including social security numbers. The review also found that hackers or others
could obtain access to technical information about the applications, potentially
providing hackers with additional avenues of attack.
Web applications contain critical vulnerabilities—Sensitive information is
exposed because all 12 of the Web-based applications auditors reviewed
contained a number of critical vulnerabilities. These flaws could potentially result
in ADE’s systems and the information contained within them being further
compromised.
Auditors identified 8 different types of critical vulnerabilities—including 4 classified
as high risk—in the 12 applications reviewed. Most of the applications contained
at least five vulnerabilities. These weaknesses could potentially:
ADE uses Web-based
applications to collect
and report information
about students,
teachers, and schools.
An independent review
found that a hacker
could access social
security numbers.
􀁺􇩅 Enable an attacker to obtain access to other users’ accounts or perform
functions for which they are not authorized;
􀁺􇩅 Enable an attacker to view sensitive files or information; or
􀁺􇩐 Provide attackers with useful information that could assist them in carrying out
further attacks.
The eight types of vulnerabilities that auditors found at ADE are considered critical
and are well-known in the Web application development and security fields.
ADE has taken some actions to address weaknesses found in its
Web-based applications—In response to the Web-based application
vulnerabilities found in this audit, ADE began to take steps to identify critical
applications and prioritize actions needed to assess and correct discovered
vulnerabilities. These actions included developing a plan to focus the efforts of its
systems development staff over a 60-day period on discovering and correcting
vulnerabilities it has identified in its critical Web-based applications. ADE
management asserts those activities were performed during May and June 2006.
ADE also developed security reference guides for developers to use when creating
and maintaining Web-based applications.
ADE has not adequately addressed important security
concerns
In addition to the security problems found with its Web-based applications, auditors
also found that ADE has not adequately addressed a number of important security
concerns. For instance, significant IT security policies and procedures are either
missing or are ineffective. Additionally, ADE has not taken sufficient action to address
known security problems raised in previous security assessments or identified by
ADE itself.
Significant security policies and procedures are missing or
ineffective—Establishing an effective set of policies and procedures to address
IT security is critical to ensuring that computer systems and the information
contained within them are properly controlled and used. However, auditors’ review
of 13 significant IT security objectives found that ADE is either missing or has
ineffective polices and procedures over most of these areas.
IT best practices guidelines and policy established by Arizona’s Government
Information Technology Agency (GITA) highlight the importance of and need to
establish and communicate policies and procedures over IT processes, including
security. For example, a GITA standard addresses the need to establish rules for
appropriate use and protection of data, including classifying data as either public
Office of the Auditor General
page 11
All eight types of
vulnerabilities are critical
and well-known in the
security field.
or confidential. This helps to ensure that agencies direct the appropriate amount
of resources to protect sensitive and critical information.
However, auditors’ review of ADE’s efforts to address a number of important
security objectives revealed that ADE has not established effective policies or
procedures. As shown in Table 1 (see page 13), while ADE had established an
adequate policy for some of the areas reviewed, most areas were missing well-defined,
complete, and appropriately implemented procedures that specified how
it would achieve those objectives. In addition, in most cases auditors found that
although ADE had procedures in place, they were ineffective in achieving the
desired objective. For example, while ADE has a policy that calls for monitoring
user access by maintaining access logs, it has not developed well-defined
procedures or appropriately implemented the policy. While ADE keeps access
logs, it reports that it does not routinely review the logs to identify unauthorized or
abnormal events. Further, in the account management area, auditors found that
ADE does not have a policy to address important issues such as regularly
monitoring all user accounts and closing them when they become inactive. While
ADE follows some account management procedures, they are not complete or
appropriately implemented. In addition, the procedures they do have contain
weaknesses that could allow someone to obtain an unauthorized account or to
have privileges inappropriate to their duties.
In order to address these deficiencies, ADE needs to identify specific security
objectives that are appropriate for its systems and users, assess its current set of
policies and procedures against those objectives, analyze any gaps that are
identified, and after considering the risk associated with each, develop a plan to
implement effective policies and procedures. ADE then needs to monitor them on
a regular basis.
ADE has not taken sufficient action to address known security
problems—Although ADE has been made aware of many security concerns
through previous security assessments, it has not always taken sufficient action to
address them. Additionally, it has not developed a coordinated or consistent
approach to ensure that it does address security concerns that it identifies or that
come to its attention.
ADE has had two external security assessments in recent years, but has often
failed to take sufficient action to address problems found. According to ADE, in
response to an attack in which a hacker defaced ADE’s Web site in 2002, ADE
contracted for a vulnerability assessment to provide a comprehensive set of
security recommendations. ADE indicated that in 2004, one of its major software
vendors provided an even more comprehensive security assessment of its
systems at no cost. Several different reports were produced as a result of the 2004
review. While ADE indicated that it has addressed the specific vulnerabilities found
in the 2002 review, it has not fully considered or addressed several important
findings from the 2004 assessment. For example, the 2004 assessment
State of Arizona
page 12
ADE does not have
effective security
policies and
procedures.
ADE has not addressed
several serious security
weaknesses identified in
2004.
Office of the Auditor General
page 13
State of Arizona
page 14
highlighted some common but potentially serious weaknesses in some of ADE’s
Web-based applications, network security, and internal processes. While ADE
did fix some specific weaknesses found in its Web-based applications, it has
indicated that other weaknesses noted in the reports were not addressed.
Several of these weaknesses were critical and should have been addressed
immediately. ADE personnel indicated that many security weaknesses have not
been addressed because they lack needed resources, because of the difficulty
in scheduling system downtime needed to address weaknesses, and because
of an agency-wide lack of focus on security.
Further, ADE has indicated that it did not use the results of its 2004 review to
assess whether its other Web-based applications had similar problems to those
found, nor did it take action to ensure that it prevents future problems by
including more focus on security as it develops new applications. For example,
ADE has not fully established guidelines to identify and address security
requirements during its software development and testing processes. In
addition, staff responsible for developing and testing ADE’s computer
applications did not feel they had been properly trained to recognize and
address security vulnerabilities.
ADE has also identified a number of security concerns on its own and has
reported these on its 2005 and 2006 Technology Infrastructure Standards
Assessment (TISA) report, which is submitted to GITA on an annual basis.1
However, it has only recently begun to take action to address some important
items. For example, ADE’s 2006 TISA indicates that it does not test its backup
data for restorability or classify its data according to risk. However, ADE has only
recently developed a preliminary plan for testing backups and does not currently
have any specific plans in place to address data classification. ADE indicates
that for many of the weaknesses noted in the TISA reports, corrective action is
either in progress or it lacks adequate resources to implement a corrective action
plan.
Finally, ADE has indicated that it has not had a coordinated approach for
addressing security concerns that it identifies or that come to its attention.
Security issues that have been addressed or corrected in the past were done by
individual initiative rather than in a coordinated and consistent way. ADE should
develop a specific process to evaluate its known security concerns, prioritize
them in order of risk, develop a plan to address them, and ensure that
responsibility for correcting them is appropriately assigned. ADE should also
develop a process to identify and implement specific security guidelines for its
systems, incorporate them within its systems development and testing process,
and train its development and testing staff on security concerns and methods.
1 The TISA is a self-assessment required by GITA to help agencies identify their IT security vulnerabilities. In 2005, it
was called the Technology Security Assessment.
ADE does not have
plans in place to classify
data according to risk.
Office of the Auditor General
page 15
ADE should consider assigning responsibility for IT
security
ADE should consider creating an appropriate position to be responsible for all IT
security within ADE. If established, this position should be charged with responsibility
for achieving the actions recommended in this report and for routinely conducting an
agency-wide assessment of IT security policies and practices to ensure that agency
security objectives are met. This would help to further raise security awareness within
ADE, establish security as an ADE priority, and ensure that someone is accountable
for addressing security-related issues and concerns.
The U.S. Government Accountability Office (GAO) found that leading organizations
establish centralized security responsibilities over information security practices.1
GAO indicated that this enabled those organizations to achieve some efficiencies
and increase consistency in the implementation of their security programs. Further,
best practices indicate that when assigning such responsibility, it should be:
􀁺􇩅 Established at the organization-wide level to enable effective actions with
organization-wide security issues and should be independent of day-to-day IT
operations; and
􀁺􇩅 Embedded at an appropriate senior level to enable effective design,
implementation, and enforcement of the organization’s security policies and
procedures.
1 U.S. Government Accountability Office. Executive Guide: Information Security Management (GAO/AIMD-98-68).
Washington, D.C.: GAO, May 1998.
Leading organizations
establish centralized
security responsibilities.
Recommendations:
1. ADE should develop and implement an ongoing process for addressing IT
security vulnerabilities or control weaknesses when they are discovered. The
process should ensure that known security concerns are evaluated and
prioritized in order of risk, that specific plans to address them are developed,
and that responsibility for correcting them is assigned.
2. ADE should identify specific security objectives, assess its current set of policies
and procedures against those objectives, analyze any gaps, consider the risk
associated with each, develop a plan to implement effective policies and
procedures, and monitor them on a regular basis.
3. ADE should develop a process to identify and implement specific security
guidelines for its systems, incorporate them within its systems development and
testing process, and train its development and testing staff on security concerns
and methods.
4. ADE should consider creating an appropriate position to be responsible for all
IT security within ADE. The reporting line of the security position should be such
that it can effectively design, implement, and enforce compliance with the
organization’s security policies, standards, and procedures, and ensure that
they are functioning effectively.
State of Arizona
page 16
ADE can further enhance SAIS’ reliability
ADE can further improve SAIS’ reliability and processing performance. SAIS
processes and maintains a large amount of valuable data, and its success requires
the cooperation of all parties involved, including school districts and charter schools
who record and submit the data, vendors who develop the majority of the software
packages used to submit data, and ADE. According to a survey conducted by our
Office, although the majority of SAIS users have confidence in the accuracy of SAIS’
data, some users continue to raise concerns. As the primary entity responsible for
maintaining and operating SAIS, ADE plays an important role and can help further
improve SAIS reliability by doing three things: (1) adding more controls to ensure
data accuracy, (2) taking basic steps to improve functionality, such as providing the
capability to archive reports, and (3) monitoring the performance of the various SMS
software packages and possibly establishing a rating or certification process.1
SAIS processes and maintains a large amount of
valuable data
SAIS collects and processes student data that is used for determining academic
progress, meeting state and federal reporting requirements, and most importantly,
according to ADE representatives, data within SAIS is the basis for calculating ADM
and distributing approximately $3 billion in state funding for education. SAIS
continuously evolves because SAIS data requirements may change from year to year
because of state education mandates. In addition to data processing within SAIS at
ADE, the operation of SAIS involves significant data processing at the district, school,
or charter school level. A.R.S. §15-1042(B) requires school districts and charter
schools to submit student-level data to SAIS. There are approximately 700 districts
and charter schools.
These entities use SMS software packages to record and process information locally
and transmit the required student-level information to SAIS.1 Once data is received
Office of the Auditor General
page 17
1 Districts, schools, and charter schools use SMS software packages to store and manage student information and submit
electronic information to SAIS.
FINDING 2
at ADE, to help ensure data accuracy, SAIS conducts three different levels of data
validation, with the first applied to the file submitted and the next two applied to
records, or transactions, within the file. In fiscal year 2006, districts and schools
submitted more than 46.5 million transactions, and according to an ADE report,
approximately 9 percent of the transactions submitted failed. Failures can result from
a number of factors, including data entry errors by school districts and charter
schools, SMS software issues, or ADE issues such as system failures. As such, the
success of SAIS requires the cooperation of all parties involved, including those who
record and submit data, ADE, and the vendors who provide a majority of the SMS
software packages used.
Concerns regarding SAIS
Some SAIS stakeholders have raised concerns about SAIS data reliability and the
additional resources required to work with SAIS both recently and in the past. For
example:
􀁺􇩄 Data accuracy concerns—Although the majority of SAIS users surveyed for this
audit have confidence in the accuracy of SAIS data, some concerns continue to
be voiced. In 2004 and again in 2005, the Arizona School Administrators
adopted a formal position that “the data within the system is unreliable. The lack
of reliability has a direct impact on both funding and academic performance.”
State of Arizona
page 18
SAIS Processing Data
(Unaudited)
Fiscal Year 2005
􀁺􇨴 43.4 million—Approximate number of total student data transactions.
􀁺􇨳 39.5 million—Approximate number of transactions successfully processed.
􀁺􇨳 3.9 million—Approximate number of transactions that failed processing.
􀁺􇨹 96,600—Approximate number of transaction failures due to system errors (i.e., caused by
software operating system, hardware failures, etc.)
􀁺􇨹 91 percent—Overall success rate of all transactions processed.
Fiscal Year 2006
􀁺􇨴 46.5 million—Approximate number of total student data transactions.
􀁺􇨴 42.4 million—Approximate number of transactions successfully processed.
􀁺􇨴 4.0 million—Approximate number of transactions that failed processing.
􀁺􇨱 145,000—Approximate number of transaction failures due to system errors.
􀁺􇨹 91 percent—Overall success rate of all transactions processed.
Source: Auditor General staff analysis of transaction information provided by the Arizona Department of Education from SAIS for fiscal years
2005 and 2006.
1 In addition to districts and charter schools submitting data to SAIS, auditors learned that district schools may also submit
student-level data to SAIS using SMS packages. According to agency information, there are approximately 1,600 district
schools in Arizona.
Similarly, 29 percent of the SAIS users surveyed for this audit reported that they
were not confident that the data in SAIS was accurate when it was needed for
final reporting or funding purposes. In addition, more than one-third of survey
respondents reported that they had experienced problems with student data
being dropped or disappearing from SAIS.
􀁺􇩒 Resource concerns—Some SAIS users expressed concerns about the time and
money required to work with SAIS. One survey respondent reported, “SAIS has
cost our district a lot of time and money. Time-consuming administrative
requirements and burdens are continually added . . . We had to add staff to
handle SAIS duties with no additional dollars.” Another survey respondent
stated that because of the complexity of SAIS and their SMS package, the
district felt the need to attain outside assistance, which cost more than $25,000.
According to the respondent, this was a “massive amount of money” for the
district. Similarly, one third-party respondent who assists several schools and
districts in working with SAIS stated, “An overriding concern of administrators is
the huge amount of staff time required for SAIS reporting . . . In a time when there
is a continual push to lower administrative costs, this is a tremendous burden.”
Concerns about the resources required to work with SAIS are not new. In 2003,
the Pima County School Board/Superintendent Collaborative sent ADE a letter
expressing similar as well as other concerns. Auditors spoke with a collaborative
representative who indicated that ADE worked hard to address many of the
issues and did so successfully. However, the representative also indicated that
SAIS is not a user-friendly system, requiring major staff efforts to ensure
accuracy, and that at this point, most have just accepted that reality.
ADE should take additional steps to improve SAIS data
reliability and processing
ADE can further improve SAIS. Although some controls exist, ADE needs to
implement additional controls to better ensure that data submitted by users and
processed through SAIS is accurate, complete, and current. In addition, ADE can
take steps to improve SAIS’ functionality by making modifications, such as providing
the capability to archive reports.
SAIS controls could be enhanced—As indicated on page 18, ADE has
established some data validation controls, and enhancing those controls would
help further ensure data accuracy. Controls are used in computer applications to
ensure complete processing and data integrity. Auditors reviewed two main types
of controls—input and process. Input controls help ensure the accuracy of data as
it is entered into the system, while process controls help ensure that no data is lost
inside the system. Both can be improved as follows:
Office of the Auditor General
page 19
􀁺􇩉 Input controls should be improved—SAIS has some types of input controls in
place, but entirely lacks one type of input control. The first type of input
controls, file checking, ensures files received from users can be processed by
SAIS. For example, if SAIS cannot accept the file because the file size is too
large or the file’s general structure is incorrect, the file is sent back to the user
to be corrected. The second type of input control, data validation, ensures a
basic level of data accuracy and completeness; for example, a gender field
being checked to contain only the values “M” or “F,” or a birthdate field being
required.
ADE lacks a comprehensive, department-wide documented process for
developing and implementing the business rules that support a third type of
input control, integrity checking. Integrity checking verifies that the data
submitted complies with business rules. According to ADE, business rules are
derived from statutory requirements and ADE policy, such as requirements
related to public education reporting or funding. Although some units, such as
the School Finance unit and the IT section, have drafted some procedures for
establishing business rules, these procedures have not been formalized.
Instituting a comprehensive, department-wide process for implementing
business rules is important since new rules are added on a regular basis, and
such a process could help ensure all appropriate business rules are effectively
established. Auditors were told of some instances where business rules were
established, but did not work as intended. However, auditors were unable to
pinpoint why this happened because ADE’s process has not been formalized
and lacks adequate supporting documentation. For example, ADE was
unable to successfully implement a business rule stating that a student must
not have graduated from the highest grade taught in the school district to be
eligible for state aid. This requirement should apply only to high school
graduates. However, the business rule was incorporated into SAIS in such a
way that it was preventing some other students from receiving funding, such
as those graduating from 8th grade. As a result, ADE needed to disable the
rule, but is working to revise and implement it for fiscal year 2007.
To help ensure appropriate business rules are completely and successfully
implemented, ADE should should establish a department-wide,
comprehensive procedure. This procedure should include all the steps from
identifying what rules are needed to implementing those rules in SAIS. This
procedure should also ensure that all appropriate parties are involved in every
stage of the process, such as the conceptualization of the rules, requirements
development, and the review and approval of testing results prior to
implementing the rule in SAIS. ADE may find that this procedure could be
developed through its Student Detail stakeholders meetings facilitated by the
System Training and Response (STaR) team. This group is working to
coordinate the interests of all ADE unit staff who work with SAIS and IT staff
are regularly invited participants.
A fourth type of input control, automated variance checks, does not exist in
SAIS and should be implemented. Variance checks call attention to unusual
State of Arizona
page 20
ADE lacks controls to
verify that all data
complies with all
statutory requirements.
Integrity checking
verifies that data
submitted complies
with statutory
requirements.
differences in data, such as a large change in the number of students from the
prior year. According to ADE’s School Finance unit, the unit currently conducts
manual variance checks for some charter schools by comparing current
charter school data to previous data used for determining funding. School
Finance unit staff report that the manual variance checks are not done to
identify errors made by the schools in submitting the data, but rather to ensure
that no SAIS problems have caused the errors. The unit does not review every
charter school’s data, but rather unit staff look for those schools that may
experience drastic funding cuts. According to the unit, the threshold
established to select the charter school data in need of further investigation is
set at a high level. However, the unit reports working within the resources and
time available to conduct the variance checks.
ADE could improve this manual process by establishing automated variance
checks that would look at the data for all districts and charter schools rather
than just certain charter schools. To determine what types of automated
variance checks would be the most valuable, ADE should ensure that
appropriate staff, such as IT staff and other ADE business units using SAIS
data, are involved in identifying, developing, testing, and implementing these
checks. An important part of implementing automated variance checks would
also include assigning responsibility for following up on any data variances
that appear unreasonable, and determining at what point or points in the
process to include these automated variance checks.
􀁺􇩁 Automated process controls should be implemented—Although some
manual process controls exist, SAIS does not have any automated process
controls. Best practices state that process controls are used to ensure that no
data is added, lost, or altered during processing. Many survey respondents
complained of data being “dropped” or “disappearing” from SAIS. Although
the data may still be somewhere in the system, it is not moving forward and
counted by ADE’s School Finance unit for school funding. This often results in
ADE’s School Finance unit having to reconcile the data manually, or requires
manual intervention from IT staff to resolve. In addition, without process
controls in place, users may or may not find the causes of inaccuracies with
their data.
ADE should add some process controls such as run-to-run totals and data
reconciliation. These types of checks, run at key points in the process, would
help ensure that when individual student-level data is uploaded or moved from
one SAIS process to the next, that no data is lost or altered. If the process
controls alert the user to potentially lost data, the problems can be found, and
the failed processes can be corrected. Once implemented, process controls
should be run and reviewed at least twice a month in an attempt to prevent
potential problems with SAIS data
Office of the Auditor General
page 21
SAIS lacks automated
process controls that
ensure that no data is
added, lost, or altered
during processing.
SAIS lacks variance
checks that call
attention to unusual
differences in data.
SAIS functionality can be improved—ADE can take some basic steps to
improve SAIS’ functionality with minimal impact to resources. ADE is in the process
of implementing one change. In addition, establishing a tactical team would help
ADE determine and prioritize other stakeholder needs.
􀁺􇩁 ADE should continue to develop and provide archived, funding-related reports
to SAIS users—Providing archived reports in SAIS would improve users’ ability
to reconcile student data and ensure that ADE calculates correct funding.
Currently, with one exception, SAIS has the ability to retain only the most
current version of reports, meaning if there is a processing problem, users
cannot obtain previous reports from SAIS to correct the situation. Having
archived reports would alleviate this problem. Auditors interviewed a SAIS
user, who described a situation where reports they saved allowed a charter
school to retain funding for students who had legitimately attended the school.
Had the issue not been resolved, ADE would have taken back all of the money
previously given to the school for the students whose records were in
question. Schools also need to compare data processed each month with
data processed from previous months to ensure student data has been
processed by SAIS. Their ability to make such comparisons would be
enhanced if SAIS included a place where past records could be archived for
retrieval at any time.
In fiscal year 2006, the IT section developed a tool for archiving one funding-related
SAIS report, but user acceptance is still needed. This report displays
the results of the most recent aggregation for average daily membership and
average daily attendance by grade level, school, and reporting period. Other
than this report, SAIS has the capability to retain only the most recent version
of reports. SAIS users can pull data reports that provide various types of
information about the data, including unadjusted counts of students at
schools, adjusted counts, and special education information. However, a user
must keep printed copies of such reports in order to refer to them at a later
time. Therefore, ADE should obtain user acceptance of the one report it has
developed for archiving, and also work to develop and implement additional
archived reports. As part of this process, ADE should ensure that appropriate
SAIS stakeholders are involved in determining what funding-related SAIS
reports should be archived, as well as in user acceptance testing.
􀁺􇩁 ADE should establish a tactical team to address its users’ needs—ADE
should establish a tactical team composed of the ADE IT section and other
internal and external stakeholders to determine and prioritize changes to SAIS
that will help address SAIS users’ concerns. The tactical team should focus on
changes that are the most feasible, such as those that will have the biggest
impact and can be done with a reasonable amount of resources. For example,
the tactical team might consider whether producing reports mid-month, which
ADE’s school finance staff believe would allow the unit and charter schools to
identify errors that need to be corrected for accurate payment calculations in
a more timely manner, is something that the SAIS community feels should be
State of Arizona
page 22
Archived reports could
facilitate schools’
month-to-month data
comparisons.
implemented. However, the tactical team would need to evaluate the impacts
and the relative benefits of stopping SAIS processing mid-month to produce
the reports relative to other needed SAIS changes the team may identify. Once
SAIS changes are identified and prioritized, the tactical team should establish
a schedule for implementing the agreed-upon SAIS changes.
ADE should take steps to improve oversight of Student
Management System (SMS) software performance
In addition to the changes needed in SAIS, ADE should improve its oversight of the
SMS software used by districts, schools, and charters to submit student data to SAIS.
Auditors’ survey of SAIS users and a review of data submission error reports
identified problems with software performance. There are several reasons SMS
software package quality varies, including ADE’s failure to monitor software
performance. ADE can encourage improvement of software performance by
ensuring its SAIS SMS software test environment is up-to-date and available,
monitoring software performance, and possibly establishing an SMS software rating
or certification process.
Many software packages used with varying levels of quality—The
numerous software packages developed to collect and transmit student
information to SAIS vary in their ability to successfully submit data. According to
agency information, at least 20 different software packages, developed either by
districts or purchased from outside vendors, are used to collect and transmit
student data to SAIS. However, auditors found that the quality of these products
varies considerably. At auditors’ request, ADE generated a report of the type and
amount of transaction errors associated with each software package submitting
data to SAIS during fiscal years 2005 and 2006. According to the report, during
fiscal year 2006, one software package had an error in 0.7 percent of the
transactions submitted to SAIS, while another software package had a 31.8
percent error rate. These errors can have various causes, including issues with
SMS programming that can result in errors when the data is correct. Regardless of
the causes, whenever these errors occur, districts, schools, and charters must
resubmit their data, and SAIS must then reprocess it, which requires additional
time on the part of the district, school, and charter staff, and additional ADE
resources.
In addition, auditors found that when users upload electronic files of student data
records to SAIS and are notified that a few records have an error, a few software
packages require that full files be sent back through SAIS instead of only the
records that have been repaired. According to ADE, this can create unnecessarily
large files, when fixing only the incorrect record or records through a change
transaction could have resolved the issue. For example, if a student were
incorrectly marked as being in first grade, but is really in second grade, a change
Office of the Auditor General
page 23
At least 20 different
software packages with
varying quality are used
to collect and transmit
student data to SAIS.
transaction would correct just the student’s grade. However, some SMS software
packages address corrections like these by deleting the student and all associated
information from SAIS, and then resubmitting the student and all other data
associated with that student. According to ADE, there is always a chance that
during this type of error correction process, something could be coded incorrectly
and the student’s data would never get added back into SAIS.
Some software packages may vary in quality because ADE has not established
clear guidelines. For example, although ADE provides vendors with information
about internal SAIS controls that check the basic accuracy and completeness of
submitted data, the agency does not require vendors to include these controls in
their software. ADE hopes that vendors will include these controls to reduce ahead
of time the amount of errors identified in data submitted to SAIS. As a result, the
five vendors that auditors interviewed varied in the amount of controls they include
in their software. For example, two vendors stated that they implement all the error
checks they can while another stated that they offer only some error checks.
Users concerned with SMS software performance—Auditors’ survey of
SAIS users identified concerns with software performance. For example, one
district who was using a particular vendor reported that they lost all of their
kindergarten data during the SMS software package’s automatic process of
updating student records to reflect the students’ change to first grade. The district
had to create a new file for the prior year so they could receive their ADM for those
kindergartners. This district has since switched to another SMS software program
and feels the software has allowed them “a better start.” Another district reported,
“We have had software issues that have taken a long time to address, and I
question why the software was not tested more thoroughly before release to
schools.” Finally, a charter school respondent provided the following feedback:
“We believe that many of our problems with SAIS data originated with the interface
between (the vendor product) and SAIS.” Although some survey respondents
voiced frustration about the vendors’ role in software performance, survey
respondents clearly felt that some of the responsibility fell to ADE because SAIS is
an ADE system.
Several reasons software quality varies—Software performance varies for
several reasons, as follows:
􀁺􇩁 ADE has relaxed software testing requirements—ADE applied more stringent
testing requirements to vendors who wished to be listed on an ADE Web list
of vendors supporting SAIS when SAIS was first established. The initial tests
required vendors to successfully submit to SAIS a variety of student data file
transactions, such as files related to student absences, student assessment
information, and Special Education program participation. Vendors were also
asked to demonstrate that their software could successfully submit data
representing a variety of cases, such as a student registering for the first time
at a school, but who also failed to attend. According to ADE, each vendor was
State of Arizona
page 24
One survey respondent
said that many of their
SAIS data problems
originated with the
vendor product and
SAIS interface.
also assigned to work with an ADE employee throughout the process. If a
vendor’s SMS software package was successful in completing all transactions
and case studies, vendors and their products were listed on ADE’s Web site
as a “vendor supporting SAIS.”
Since the initial round of software testing, however, ADE has relaxed its
requirements. According to ADE, resource limitations have prevented it from
continuing the original process. However, it still requires that vendors
demonstrate that their software product is able to successfully submit the
required student data file transactions. In addition, the vendor must provide a
letter or e-mail from a client indicating that they are using the product and are
satisfied with it. However, ADE no longer requires that new software products
successfully complete the case studies, and it does not assign a specific ADE
employee to work with the district and/or vendor throughout the process.
According to ADE, only two software products have been added to the list of
vendors after SAIS was implemented. However, because ADE does not
require vendors to annually demonstrate that their product will work with the
most current version of SAIS, one of the newer vendors was listed by ADE on
its Web site even though it had only shown that its product worked with the
previous year’s version. This occurred even though the current version of SAIS
had incorporated a number of changes.
􀁺􇩁 ADE’s test environment is not always current—The test environment ADE
developed for vendors and districts to test their SMS software products on is
not always ready when needed. As a result, the vendors and districts
developing their own SMS software sometimes do not have sufficient time to
correct any software problems they may find before the software needs to be
used to actually submit data to SAIS. For example, for the 2005-2006 school
year, ADE’s test environment was not available until after the school year
began. As a result, one vendor reported having some of its clients use its
untested software to submit their actual data to SAIS so that it could identify
whether errors occurred and then correct the problems before providing an
updated version of the software to its other clients. Finally, another vendor
reported having to correct problems with its SMS software that would have
been caught before its clients began using it if the test environment had been
available.
According to ADE, the test environment was not available this year because
SAIS system changes were not ready and there was not a server available for
SMS software testing. ADE also indicated that keeping SAIS up and running
takes priority over maintaining the test environment. Specifically, SAIS
changes are implemented in SAIS and the SMS software test environment at
the same time, and ADE runs tests to make sure that the changes were
successful in SAIS. However, when new changes result in problems in SAIS as
well as the SMS software test environment, fixing SAIS takes priority.
Office of the Auditor General
page 25
Districts and vendors
do not always have
sufficient time to test
and correct problems
before their SMS
software needs to be
used.
􀁺􇩁 ADE does not monitor or provide assurance regarding software
performance—ADE fails to monitor SMS software packages after they are
placed on the ADE list of vendors supporting SAIS and does not provide
assurance that the listed software packages are successfully able to meet
SAIS processing requirements. As previously mentioned, at auditors’ request,
ADE generated a report showing the number and type of transaction errors
experienced by each software package. Auditor analysis of the report
identified numerous, specific transaction types, such as those related to
support program participation (i.e., submission of information related to
participation in programs for block grants), and student assessments (i.e.,
data concerning assessments administered to a student in order to identify a
language need), which experienced a failure rate of 33 percent or more by at
least three different software packages. This report also indicates that most
problems are associated with vendor-developed software products rather
than software developed by the districts.
Because ADE does not monitor performance, the agency does not assure
SMS software performance, even though these SMS software packages may
be listed on its Web site. Instead, the agency has added a caveat to the Web
page listing vendors supporting SAIS stating that there was no way for the
agency to verify that all transactions and files generated by the vendor’s SMS
software package were submitted to ADE without alteration. However, auditor
analysis of the ADE-generated software package error report found that the
majority of SMS software packages listed on ADE’s Web page have problems
submitting some transactions, regardless of when the SMS software was
developed.
ADE should take actions to improve SMS software performance—
ADE could take action to facilitate improvement of SMS software performance.
Although ADE indicated it does not have the authority to ensure SMS software
compliance, auditors found that ADE has adequate authority. Specifically, A.R.S.
§§15-901(A)(15) and 15-1042(A) require ADE to prescribe the format, manner, and
procedures for electronic data submission. Auditors identified three types of
actions ADE should take to improve software performance, as follows:
􀁺􇩅 Ensuring the test environment is updated—ADE should ensure that the test
environment is up to date and available for use at the beginning of the school
year and at other times as needed so vendors and districts developing their
own SMS software can demonstrate that their software can successfully
submit the required student data file transactions to SAIS.
􀁺􇩄 Developing a proactive means of monitoring vendor performance—ADE
should use transaction error reports, such as those generated at auditor
request, to monitor and analyze software performance, identify potential
problems, and perhaps provide increased assistance to vendors as a means
to help identify and resolve problems with particular transactions. Monitoring
and following up on identified software problems may also help ADE reduce
State of Arizona
page 26
Monitoring and following
up on identified software
problems may help ADE
reduce the number of
files that SAIS must
reprocess.
the number of files that SAIS must reprocess. ADE may also use these reports
as a preliminary step to establishing software package performance
standards for rating or certification.
􀁺􇩒 Rating or certifying software packages—ADE should consider rating or
certifying software packages. Iowa’s Department of Education certifies
Student Information System (SIS) vendors whose products are used to submit
data to Iowa’s Department of Education. Iowa certifies vendor products twice
a year and has different processes for returning and new vendors. Auditors
interviewed five vendors providing SMS software packages to clients in
Arizona. All five supported some type of a certification or compliance process,
although some were more supportive of the concept than others. To further
ensure that SMS software packages comply with SAIS requirements, ADE
should work with SMS software vendors and users to determine what factors
should be included in the process, how frequently the process should occur,
and where to publish the results of the certification or rating process, such as
on their Web site. ADE would also need to establish a mechanism for
addressing vendors who do not meet the recurring certification or rating
requirements. Any rating or certification process that is implemented should
allow sufficient time for software developers to meet the requirements.
Office of the Auditor General
page 27
Recommendations:
1. To improve SAIS data reliability, ADE should implement additional controls.
Specifically, ADE should:
a. Establish a department-wide comprehensive procedure for developing and
implementing business rules;
b. Implement automated variance checks by identifying appropriate staff to
determine what types of variance checks should be added, as well as
assigning responsibility for following up on any data variances that appear
unreasonable; and
c. Add processing controls such as run-to-run totals and data reconciliation,
and review information collected from the controls at least twice a month to
help prevent potential problems with SAIS data.
2. To help improve SAIS’ functionality, ADE should:
a. Obtain user acceptance of the one report that has been developed for
archiving, and
b. Develop and implement other SAIS-archived reports.
3. To address user concerns and identify additional ways to improve SAIS, ADE
should:
a. Establish a tactical team composed of representatives from ADE’s IT
section and both internal and external stakeholders to identify and prioritize
its user community’s needs, and
b. Establish a schedule for implementing the agreed-upon SAIS changes.
4. To improve SMS software performance, ADE should:
a. Ensure the SAIS SMS software test environment is up-to-date and available
when needed;
b. Monitor software performance and take steps to address any problems
identified; and
c. Consider establishing a recurring SMS software certification or rating
process.
State of Arizona
page 28
Office of the Auditor General
page 29
ADE needs to improve IT project management
and operation oversight
ADE needs to improve its approach for managing systems development projects
and for overseeing its IT operations. ADE’s environment for developing IT projects is
currently characterized by a lack of consistency, with projects being developed both
inside and outside the IT section and without a clear and consistent development
approach in either setting. Documentation explaining how a system functions and
how to use it is often missing or of limited quality. ADE has taken some recent steps
to provide direction and guidance over development activities, but additional work is
needed. To ensure effective IT operations, ADE needs to (1) adopt a more structured
process for developing and maintaining its IT systems and (2) improve key IT
oversight activities, such as regularly monitoring key performance measures,
conducting risk assessments, and maintaining and testing a business continuity
plan.
IT project management approach lacking
The IT section at ADE develops IT systems ranging from relatively small applications,
such as one called the Arizona State Tutor Fund, which is used to collect data for the
Failing Schools Tutor Fund program, to much larger and more complex applications
such as SAIS, which is used to collect state-wide, student-level data and to perform
a variety of data aggregation and reporting functions that are critical to calculating
state funding for education. ADE has had a recent history of ad hoc development of
information systems, and that approach has resulted in several inconsistencies.
These inconsistencies can be seen in where and how systems are developed, as
well as in the documentation prepared to explain and support the systems.
Information systems development scattered and unstructured—
Several of ADE’s current information systems have been built by various
development teams who in most cases have not followed a standard agency
development process, resulting in information systems that are difficult to maintain
FINDING 3
and use. IT management and staff members agree that the information systems
development process within the IT section varies by team and by project. For
example, according to IT developers and the Director of Requirements and
Testing, the majority of changes made to SAIS go through the IT Quality Assurance
Team (QA) before being put into production, but changes to other applications
have traditionally not gone through a formal QA process. Instead, according to one
IT developer who works on the Grants Management application, changes are
usually only informally tested by users before being fully implemented. Another
example comes from the Exceptional Student Services (ESS) unit that has an IT
developer who works on all of the applications and databases that the ESS unit
uses. The developer has built and currently maintains several important ESS
information systems that are used for meeting federal reporting requirements, but
does not follow a specific development process and has created minimal
documentation of how the systems operate.
Further, software applications have been developed outside of the IT section,
again without a standard agency development process. According to IT
management, these projects sometimes fail or develop problems that the IT
section is then requested to provide assistance in correcting and/or with
maintaining the information system. For example, the Certification unit is currently
developing a “Teacher Certification Web System.” According to the agency, this
project has encountered many problems, including poor vendor relations, and at
least part of the application is not usable and needs to be completely replaced.
ADE’s IT unit management is aware of its minimal use of a systems development
process and has acknowledged a need to improve. It has set a goal to correct
many of the deficiencies by 2008, but does not have a documented plan for how
it will reach its goals.
Key information system documentation is missing or insufficient—
Creating documentation is an important part of each phase of a good information
system’s development process. However, many of ADE’s information systems
lack good documentation, which has caused difficulties in maintaining and
modifying the systems. ADE needs to take a number of steps to address these
problems, such as identifying documentation gaps and adopting a plan to
address them.
Documentation explains how a system functions and how to use it. Good
documentation is necessary to provide easy-to-read instruction manuals for the
system’s users and to provide technical details that IT staff need to be able to
maintain and modify the system in the future. However, since ADE has not used
an adequate development process, many of the information systems that ADE
created are lacking high-quality documentation. For example, a staff member who
works on a system requirement’s guide for part of SAIS does not have a complete
data dictionary or flowchart to work with. Data dictionaries and flowcharts are
important pieces of documentation that provide a description of what the data is
and how it moves throughout an information system. As a result, if this staff
State of Arizona
page 30
IT staff do not always
have or maintain high-quality
documentation
for ADE’s applications.
The IT section does not
oversee all software
development.
Some software changes
are implemented
without going through
quality assurance
testing.
member leaves, the next person assigned to work with this system will likely face
problems understanding how the system stores and processes data. Further, IT
staff often do not keep the existing documentation updated. In the case of one
application that auditors reviewed, the programmers stated that they were not sure
which of the original documentation was still relevant, and when they made
program changes, they did not update the design documentation.
The lack of documentation of some applications has resulted in steep learning
curves for IT staff who must modify and maintain these applications. This is
especially true when, due to turnover or other reasons, staff must work on
applications they are not familiar with. For example, an IT staff member stated that
he had to create his own documentation when he was assigned to maintain an
application he had not previously worked on. The staff member described how he
had to develop his own way of understanding the application; for example, he
interviewed the outgoing developer, experimented with the application, and
reviewed the code to become familiar with how it operates. Missing or poor
documentation also leads to inefficient use of staff time. For example, the staff
member mentioned above spent a day reviewing the application to understand a
function that may have taken minutes to understand had he had good
documentation.
Lack of an adequate development process also causes inconsistencies in the way
that documentation is managed. For example, despite the challenges the staff
member mentioned above faced due to the lack of documentation, he has not
taken all of the steps necessary to ensure that the documentation he developed
will be available to a different staff member in the future.
ADE needs to take a number of steps to address its documentation problems.
First, ADE should review its current applications’ technical and user
documentation to determine what is needed to properly maintain its applications.
Second, it needs to prioritize the gaps between existing and desired
documentation. Finally, it needs to develop a plan to address the gaps and
schedule improvement activities to address them.
ADE should establish effective process for developing
information systems
ADE needs to institute an effective Systems Development Life Cycle (SDLC) process
for developing information systems. Establishing an SDLC process will help ensure
that systems are adequately developed, tested, and documented before being put
into operation. Although ADE has recently taken steps to improve its development
process, several additional steps are needed.
SDLC methodology helps ensure proper information systems
development—An SDLC process is an accepted approach used to consider
Office of the Auditor General
page 31
and develop an information system or to make a major modification to an
information system. SDLC methodologies frequently group similar activities into
phases. Typical phases are shown in the colored textbox, and each phase
typically results in a number of key deliverables.
Using an SDLC process helps ensure that systems developed meet IT mission
objectives, are compliant with the current and planned laws and regulations, and
are easy to maintain and cost-effective to enhance. An SDLC process also
provides other benefits when creating information systems, such as reducing
risks and errors, helping to ensure that security requirements are integrated into
applications and systems, and avoiding inefficient activities, such as recreating
existing components for each project.
Additional steps needed to establish SDLC process—Although
ADE has recently taken some steps to provide direction and guidance over IT
development activities, additional work is needed. ADE needs to expand and
further develop the project development guide that was recently published by the
IT section. Further, ADE needs to ensure that all IT projects, whether developed
by the IT section or other units within ADE, follow this SDLC process:
􀁺􇩅 Expanding project development guidance—In 2005, a Project Management
Office (PMO) was created to help direct technology activities. The Project
Management & Software Development Methodology guide, which the PMO
first published in November 2005, outlines an information systems
development methodology. This document provides a good start, but it needs
more detail and a better plan to ensure that it is effectively implemented and
that its use is monitored to ensure that it is properly followed. For example, the
guide is lacking standard life cycle phases (operation, maintenance, and
disposal); does not provide a plan for frequent stakeholder communication;
does not integrate security into development; and does not provide detailed
documentation templates. In addition, the methodology contained in the
guide has not been widely promoted and is not widely used. A PMO director
acknowledged that the guide needs to be updated and improved.
􀁺􇩅 Ensuring more uniform procedures throughout ADE—ADE also needs to
ensure that all projects follow a standard SDLC. Currently, the IT section does
not oversee all system development. As discussed previously, various
divisions have their own IT employees who work on projects specific to their
divisions, but do not have to follow the SDLC methodology the PMO
promotes. According to IT management, this has not only led to problems in
developing some projects, but created additional difficulties when the IT
section has later had to maintain and operate the information systems. ADE
should develop, adopt, and enforce the use of a single, effective, agency-wide
SLDC process.
State of Arizona
page 32
Systems Development Life
Cycle Phases and Key
Deliverables
Initiation
􀁺􇩎 Needs determination:
—Perception of need
—Linkage of need to
mission and performance
objectives
—Assessment of alternatives
to capital assets
—Preparing for investment
review and budgeting
Acquisition / Development
􀁺􇩆 Functional statement of need
􀁺􇩆 Feasibility study
􀁺􇩒 Requirements analysis
􀁺􇩁 Alternatives analysis
􀁺􇩃 Cost-benefit analysis
􀁺􇩓 Software conversion study
􀁺􇩃 Cost analysis
􀁺􇩒 Risk management plan
􀁺􇩁 Acquisition planning
Implementation
􀁺􇩉 Installation
􀁺􇩉 Inspection
􀁺􇩁 Acceptance testing
􀁺􇩉 Initial user training
􀁺􇩄 Documentation
Operations / Maintenance
􀁺􇩐 Performance measurement
􀁺􇩃 Contract modifications
􀁺􇩏 Operations
􀁺􇩍 Maintenance
Disposition
􀁺􇩁 Appropriateness of disposal
􀁺􇩅 Exchange and sale
􀁺􇩔 Transfer and donation
􀁺􇩃 Contract closeout
Source: Auditor General staff analysis of NIST
Special Publication 800-64 Rev. 1.
ADE should improve key oversight activities
Improving the IT oversight framework will help ADE management ensure that it can
sustain operations and implement the strategies required to extend activities into the
future. In particular, ADE’s efforts need to address three important pieces of an IT
oversight framework:
􀁺􇩐 Performance measurements—ADE needs to identify, develop, and proactively
monitor key IT performance indicators. According to IT management, the IT
section performs minimal monitoring of performance and of some other key
indicators such as server statistics or system capacity used. ADE’s IT section
should develop and proactively monitor these and other performance
measurements, such as the number of production problems per application that
are causing visible downtime, the number of IT calls handled by the Support
Center, and the number and status of requests for systems modifications. The
IT section should also monitor measurements that indicate the performance of
the section, such as the percentage of users satisfied with functionality of
systems delivered and the availability, completeness, and accuracy of user and
operational documentation.
􀁺􇩒 Risk assessments—To ensure IT risks are properly identified and mitigated,
ADE’s IT section should regularly perform IT risk assessments and develop
procedures to address issues raised. Possible risks include such things as a
hacker disrupting ADE’s Web site, the loss of key personnel, or computer
hardware failures. ADE has not adopted a framework to properly analyze and
manage such risks. Although ADE developed a policy statement that indicates
a risk assessment should be conducted twice a year, the policy is not in
operation. ADE indicates that it does not have adequate resources to fully
implement its policy. While some activities, like independent assessments, may
require additional resources, other activities, such as a basic internal
identification and ranking of problem areas, could be performed with minimal
resources. Finally, ADE does not have an effective process in place to address
issues that may be uncovered through risk assessments. As a result, even if risk
assessments were performed and issues identified, there is no assurance that
the issues would be addressed.
􀁺􇩄 Disaster recovery and business continuity planning—ADE’s IT section should
fully develop a business continuity plan that is updated and tested regularly. The
need for providing uninterrupted IT services requires developing, maintaining,
and testing IT disaster recovery and continuity plans, providing for off-site
backup facilities, and establishing regular plan training and testing. ADE’s IT
section has produced a state-required business continuity plan, but based on
auditor analysis, the plan is not adequate to address a major disruption to IT
services, and according to IT staff at ADE, the plan has never been tested. The
Office of the Auditor General
page 33
IT does not have a
method for proactively
identifying and
addressing performance
problems or
vulnerabilities.
plan is missing some key information, such as current contact information for
key personnel, contact information for vendors, and information about alternate
locations for command centers, backup sites, off-storage sites, and restoration
sites. An effective business continuity plan reduces the likelihood and impact of
a major IT service disruption on key organizational functions and processes.
Recommendations:
1. ADE should develop, adopt, and enforce the use of a single, effective agency-wide
SLDC process.
2. ADE should create a plan to review current applications’ technical and user
documentation:
a. Determine what needs improvement in order to maintain the applications;
b. Address identified gaps; and
c. Prioritize and schedule improvement activities.
3. ADE should identify, collect, and measure performance measurements for key
IT functions and operations.
4. ADE should develop a plan and address resource requirements to allow it to
perform regular risk assessments of its IT systems and operations, and should
develop procedures to address issues raised.
5. ADE should fully develop a business continuity plan and should include
provisions for regularly updating and testing the plan.
State of Arizona
page 34
An effective business
continuity plan can help
reduce the impact of a
major IT service
disruption.
Office of the Auditor General
page 35
ADE needs to ensure its information technology
meets its business needs
ADE needs to take several actions to better ensure that its information technology
efforts can meet its business needs. Information technology is key to many of ADE’s
administrative functions and responsibilities, but the IT section’s staff are not included
in the strategic planning processes of ADE or its divisions. This lack of coordination
limits ADE’s ability to ensure that initiatives will have the necessary IT support.
Instead, key initiatives may languish, or ADE units may feel it necessary to develop
IT applications on their own. Three things are needed: a steering committee to help
guide information technology efforts, a review of whether the IT section is correctly
positioned within the organization, and a more meaningful strategic planning
process within the IT section itself.
Coordination of IT efforts limited
ADE uses information technology in many of its operations, such as collecting
information for determining the amount of funding school districts and charter
schools receive. Although ADE’s strategic planning efforts are a primary way of
helping the organization determine how best to meet its responsibilities, auditors
found that IT considerations are not effectively integrated into these planning efforts.
Better coordination involves both creating a coordinating mechanism and
reassessing the IT section’s place in the organizational structure.
ADE lacks key processes to help ensure IT meets business needs—
Two key ways to help ensure that IT efforts meet ADE’s business needs are the
agency’s strategic planning efforts and its prioritization of IT initiatives. In both
cases, there are problems:
􀁺􇩂 Breakdowns in strategic planning efforts—IT management or staff do not
adequately participate in developing ADE’s strategic plan despite the fact that
IT resources may be needed to ensure objectives are met. For example, an
ADE uses IT for many of
its operations, but does
not involve IT when
developing goals that
require IT efforts and
resources.
FINDING 4
objective listed in ADE’s strategic plan calls for the delivery of high-quality
customer service by “increasing the use of data and information technology
as a management tool to make better informed decisions.”1 However, IT was
not involved in developing this objective, and the plan does not address how
IT would meet this objective or the resources that are needed. Similarly, IT
management or staff do not participate in the process that the divisions use
to develop and implement their strategic planning goals. Divisions are
assigned goals that have been developed by the Superintendent of Public
Instruction, and ADE management is given the responsibility to ensure that
related strategic plan objectives are developed and met. However, there is no
process in place for effectively involving IT when the divisions develop goals
for which IT efforts and resources are required. As a result, IT is frequently not
involved with the division’s planning efforts.
􀁺􇩎 No process for prioritizing IT initiatives—Further weakening ADE’s ability to
ensure that IT meets it business needs, ADE lacks a department-wide process
for prioritizing IT initiatives and other efforts to ensure that agency-wide and
division strategic goals are met. Rather, decisions such as how to prioritize IT
projects or requests are often made by the IT section without input from ADE
upper management and/or the affected divisions. While the IT section has
some mechanisms for recording IT requests, IT management primarily
determines how to prioritize those requests on its own.
Without sufficient planning processes in place, critical business needs may not be
met or the divisions may feel it necessary to develop their IT applications without
involving the IT section. For example, to avoid a reduction in federal funding for
NCLB, the Academic Achievement division needs to develop an IT application that
collects data on highly qualified teachers. This need has been brought to the IT
section’s attention; however, IT management believes that because of the lack of
funding, it is unable to dedicate IT staff to develop and implement this application.
In other instances, divisions or program units have hired their own programmers
to obtain the applications they need. For example, the Exceptional Student
Services (ESS) unit needed a new application to meet federal reporting
requirements for special education services. As a result of this not being an IT
priority, ESS reported that it hired its own programmer to develop this application.
ADE should take steps to ensure that the IT section effectively meets
ADE’s business objectives—ADE needs to take at least two steps to
ensure IT can effectively support its business goals and objectives. First, ADE
should establish an IT steering committee that should include representatives from
senior management, user management, and the IT function, such as the
superintendent and/or deputy superintendent, the associate superintendents, the
CIO, and other key ADE stakeholders, as appropriate. Best practice guidelines
indicate that management and other key people who understand the business
needs should be involved during the strategic planning process to help effectively
communicate project goals to project teams and the value of IT to all stakeholders.
State of Arizona
page 36
ADE needs an IT
steering committee to
help establish IT
priorities department-wide.
Without a sufficient
planning process in
place, critical business
needs may not be met.
1 Arizona Department of Education Strategic Plan, fiscal years 2007-2011.
Office of the Auditor General
page 37
An IT steering committee helps to facilitate this communication and to better
prioritize IT initiatives and projects on an organization-wide basis. This committee
should ensure adequate IT involvement in the department and division planning
processes. In addition, the committee should better direct the uses and allocation
of IT resources (people, applications, technology, facilities, and data) throughout
ADE. The committee should provide overall IT direction, help establish IT priorities
department-wide, and ensure that adequate processes exist for identifying,
funding, and allocating department-wide IT resource costs.
Second, ADE should consider whether the IT section’s placement within the
organization adequately ensures that IT can effectively meet ADE’s business goals
and objectives. The IT section reports to the Associate Superintendent of Education
Policy, and therefore is three levels below the State Superintendent of Public
Instruction, who is responsible for providing ADE’s overall direction. Best practice
guidelines indicate that the placement of the IT function in the overall
organizational structure should be contingent on the importance of IT within the
agency. Given IT’s criticality, ADE should consider whether the IT section should
be elevated within the organization. For example, in the Departments of Economic
Security, Revenue, and Administration, the CIO is placed one level higher and
reports directly to the deputy director.
IT section’s planning process ineffective
The IT section’s internal planning process is limited and ineffective. Therefore, ADE
should ensure that the IT section effectively plans for ADE’s current and future IT
needs by having IT develop an effective planning process.
IT section lacks an effective planning process—The IT
section’s current internal planning process is very limited and does not
identify the resources it needs to maintain and operate its current
systems or plan for future IT initiatives. Further, IT management is
unaware of how its plan integrates into ADE’s strategic plan, and
therefore has not ensured that its efforts are aligned with ADE’s
business needs.
The IT section’s current plan was developed to comply with a statutory
requirement that all agencies annually submit an IT strategic plan to
GITA. As a result, the IT plan is focused primarily on the information
required by GITA and does not address ADE’s strategic objectives. In
fact, according to the individual responsible for compiling the plan, the IT section
did not solicit any stakeholder input from outside the IT section when it developed
the plan. Further, the plan has remained virtually unchanged for the past 2 years
and contains four objectives that are no longer applicable. For example, the fiscal
year 2006 plan, which was finalized in September 2005, included objectives to
organize and hold conferences/workshops on various topics related to the SAIS.
However, effective in July 2005, the responsibility for this function was moved to the
School Finance unit.
Agency IT plans facilitate the
application of information
technology to enable business
initiatives, goals, and objectives of
the budget unit in an efficient
manner by describing a direction for
current and future activities,
supported by underlying principles,
standards, and best practices
Source: Obtained from GITA State-wide Policy, p. 136.
State of Arizona
page 38
ADE should ensure that the IT section develops an effective
planning process—According to GITA standards, the IT section’s planning
process should define long-term IT direction that aligns with ADE’s business
needs, and include key stakeholders such as those who utilize IT services and
information. Including key stakeholders will ensure that IT initiatives support ADE’s
mission and improve communication between IT and users on how their efforts will
add value to their services. If ADE establishes the IT steering committee proposed
earlier, this committee can provide the key stakeholder input for the IT section’s
plan.
Further, once a plan has been developed, strategies and steps to accomplish the
plan are needed. The Governor’s Office of Strategic Planning and Budgeting has
developed components of an effective strategic planning process that specifically
address the need to develop action plans to spell out the details and methods, or
strategies, that will be used to accomplish the objectives, goals, and mission of the
agency and its programs. To ensure that ADE’s business needs are being met, IT
management should develop an action plan that first formulates strategies;
evaluates costs, benefits, and possible consequences of alternative courses of
action; considers the resources needed; assigns responsibility for implementation;
defines the steps that must be finished to complete the plan; sets a time frame for
completion; and determines the resources necessary to carry it out.
ADE’s IT plan process
should include long-term
IT direction and
action plans for
accomplishing goals
and objectives.
Recommendations:
1. To ensure that IT can better meet ADE’s mission and business needs, ADE
should establish an IT steering committee that should include representatives
from senior management, user management, and the IT function, such as the
superintendent and/or deputy superintendent, the associate superintendents,
the CIO, and other key ADE stakeholders as appropriate.
2. Once established, the ADE IT steering committee should:
a. Ensure adequate IT involvement in the Department’s and the divisions’
planning processes;
b. Provide overall IT direction for the Department; and
c. Ensure that adequate processes exist for identifying priorities and funding,
and allocating department-wide IT resource costs.
3. To ensure that IT can effectively meet ADE’s business needs, ADE should review
the IT section’s organizational placement within ADE.
4. ADE should ensure that the IT section establishes an effective planning process
which includes:
a. Developing, with input from key stakeholders, an IT plan that defines the
long-term direction that aligns with ADE’s business needs; and
b. Developing an action plan that first formulates strategies; evaluates costs,
benefits, and possible consequences of alternative courses of action;
considers the resources needed; assigns responsibility for implementation;
defines the steps that must be finished to complete the plan; sets a time
frame for completion; and determines the resources necessary to carry it
out.
Office of the Auditor General
page 39
State of Arizona
page 40
ADE not in full compliance with student-level data
collection notification and disposal requirements
During the audit, auditors also identified two areas in which ADE was not fully
complying with statutory requirements. The first is a requirement that ADE tell school
districts and charter schools the specific statutory authority for collecting each item
of student-related data that ADE requires districts and schools to submit. ADE
currently is not providing this information. The second is a requirement that ADE
adopt guidelines to remove outdated student-level data collected by school districts
and charter schools from SAIS. ADE has not done so, even though statute required
it to adopt such guidelines by the beginning of the 2004-2005 school year.
ADE has not fully complied with student-level data collection
requirements—A.R.S. §15-1042 establishes data submission requirements
and timelines for SAIS. Subsection A of that statute requires ADE to “distribute a
list of the specific student-level data elements that school districts and charter
schools are required to submit.” According to ADE, it has complied with this by
providing a list of SAIS data elements on its Web site. However, subsection E
further requires that “(e)ach student-level data element shall include a statutory
reference to the law that necessitates its collection.” During the course of this audit,
auditors determined that while ADE publishes a list of the data elements contained
in SAIS, it does not include information on the statutory reference requiring its
collection. Instead, it includes information on whether the data elements are
required by the state or federal government. In order to fully comply with the law,
ADE should publish in one document both the specific student-level data elements
that it collects and the statutory reference requiring their collection. Additionally,
while there may be other data elements collected for which there is no specific
statutory reference, but nevertheless are important in order for SAIS to operate
effectively, ADE should also include those elements in its list and could cite the
reference for the process that ADE must perform, or otherwise describe how the
data fills a need in the process in order to make SAIS complete. According to ADE,
it has begun to compile this information and anticipates it will be publishable by
the middle of fiscal year 2007. ADE should finish compiling the list of data elements
Office of the Auditor General
page 41
FINDING 5
ADE needs to publish a
list of student-level data
elements that includes a
statutory reference to
the law necessitating
collection.
State of Arizona
page 42
collected through SAIS and establish a deadline to publish the information no later
than the end of fiscal year 2007.
ADE has not adopted guidelines to remove outdated SAIS student
data—A.R.S. §15-1042(I) also requires ADE to “adopt guidelines to remove
outdated student level data collected by school districts and charter schools from
the student accountability information system beginning in the 2004-2005 school
year.” Although the Legislature added this requirement in 2003, ADE has not yet
adopted any guidelines to comply with this requirement. ADE explains that it has
not adopted guidelines because changes in state and national education data
collection trends, including initiatives from the U.S. Department of Education and
Arizona’s Governor, suggest that ADE may need education data for longer periods
of time than statute currently allows. However, it appears that statute does not
prescribe a retention period for student-level data. Therefore, ADE should adopt a
retention schedule for such data and adopt guidelines to remove the outdated
data in SAIS in accordance with state statute.
Recommendations:
1. ADE should finish compiling the list of data elements that it collects through SAIS
and should:
a. Include references to the statutory authority for gathering each piece of
information, as required by A.R.S. §15-1042(E);
b. Include in its list other data elements collected for which there is no specific
statutory reference and cite the reference for the process that ADE must
perform, or otherwise describe how the data fills a need in the process in
order to make SAIS complete; and
c. Establish a deadline to publish the information no later than by the end of
fiscal year 2007.
2. ADE should adopt a retention schedule and guidelines to remove outdated
student data from SAIS.
Since statute does not
prescribe a retention
schedule for student-level
data, ADE should
adopt one.
A file containing a list of the requests or actions that computer users take or
make on a computer system or device. Sometimes referred to as “raw data,” access logs can
often be analyzed and summarized by another program to identify anomalies and/or trends.
The process of determining whether someone or something is, in fact, who
or what it is declared to be. In private and public computer networks (including the Internet),
authentication is commonly done through the use of logon passwords. Authorization
generally follows authentication.
The process of granting access to a user and allowing him/her to use the pre-approved
resources in the system, and the privileges of use (such as access to file
directories, hours of access, amount of allocated storage space, etc.). Authorization is
preceded by authentication.
The ability to monitor and measure in real time the performance of
a computer environment and to forecast its future usage. Used to determine the system
capacity needed to deliver specific operational and performance levels through quantification
and analysis of present and projected workloads.
The way a system is set up, or the assortment of components that make up
the system. Configuration can refer to either hardware or software, or the combination of both.
When you install a new device or program, you sometimes need to configure it, which means
to set various switches and jumpers (for hardware) and to define values of parameters (for
software).
A small text file of information that certain Web sites attach to a user’s hard drive
while the user is browsing the Web site. A cookie can contain information such as user ID,
user preferences, archive shopping cart information, etc. Cookies can sometime contain
personally identifiable information.
A collection of electronic information organized in such a way that a computer
program can quickly select and/or process desired pieces of data. A database is commonly
thought of as an electronic filing system.
Office of the Auditor General
page a-i
GLOSSARY
A special program used to identify coding errors or problems (bugs) in other
programs. A debugger allows a programmer to stop a program at any point and examine and
change the values of variables.
The translation of data into a secret code that is typically not in a humanly
readable form. Encryption is often an effective way to achieve data security. To read an
encrypted file, you must have access to a secret key or password that enables you to decrypt
it. Unencrypted data is called plain text.
A slang term for a computer enthusiast who enjoys learning programming
languages and computer systems and can often be considered an expert on the subject(s).
The negative connotation of hacker refers to individuals who gain unauthorized access to
computer systems to steal data and/or to carry out other ill-intended purposes.
A private, internal Web site belonging to an organization and generally only
accessible by an organization’s members, employees, or others with authorization.
Program code added, changed, or removed from a software system in
order to intentionally cause harm, perform unauthorized activities, or use up resources of a
computer or information system. Three typical types are as follows:
􀁺􇩁 A program which, when executed, can add itself to other programs without
permission and in such a way that the infected program, when executed, can cause ill-intended
actions to occur or add itself to still other programs.
􀁺􇩁 A virus that replicates itself by resending itself as an e-mail attachment or as part
of a network message.
􀁺􇩁 A program that masquerades as a legitimate program, but does something
other than what was intended.
The process of removing data from electronic storage media such that
data recovery using known techniques is prevented. It is intended to provide reasonable
assurance, in proportion to the confidentiality of the data, that the data may not be retrieved
and reconstructed.
Small, portable computing devices that allow users to store, organize, and
access a variety of information. They are typically powered by batteries and need to be
charged for use. They range in size from a stack of business cards to a paperback book.
Typical examples of mobile devices include Personal Digital Assistants (PDA), cell phones,
and laptop computers.
State of Arizona
page a-ii
Glossary (cont’d)
A computer program or a sequence of instructions that are interpreted by another
program. A script is normally used to automate routine, complex, or advanced features or
procedures within a system. Scripts are commonly used to process user information from
Web pages.
Piece of software code that can be applied after a software program has
been installed to correct an issue(s)—specifically related to the prevention of, or protection
against, access to information or systems by unauthorized recipients, and intentional but
unauthorized destruction or alteration of that information—with the original program. This can
range from fixing software bugs to improving the usability or performance of a previous
version of a program.
A flaw or weakness in a system’s design, implementation, or operation and
management that could be exploited to violate the system’s security policy. These
weaknesses could be exploited to gain unauthorized access to information or disrupt critical
processing.
A preventive me