Performance audit, Department of Corrections, contracts management function

              PERFORMANCE AUDIT
DEPARTMENT OF CORRECTIONS
CONTRACTS MANAGEMENT FUNCTION
Report to the Arizona Legislature
By the Auditor General
July 1986
86- 4
DOUGLAS R NORTON. CPA
AUDITOR GENERAL
STATE OF ARIZONA
OFFICE OF THE
AUDITOR GENERAL
July 7, 1986
Members of the Arizona Legislature
The Honorable Bruce Babbitt, Governor
Samuel A. Lewis, Director
Department of Corrections
Transmitted herewith i s a report of the Auditor General, A Performance
Audit of the Department of Corrections, Contracts Manapement Function.
This report is in response t o the July 26, 1905 resol~ rticn of the Joint
Legislative Oversight Commi ttee.
The report addresses deficiencies in the Department's contracting
process. We found that the Department has failed to comply consistently
w i t h the Procurement Code. This has led to some contract irregularities.
We also found that the Department performs limited monitoring and
evaluation of contracted programs. As a result, the Departrxent cannot
adequate1 y oversee contracted programs nor determine program
effectiveness. Finally, we found that the Department needs t o improve i t s
financial oversight and control sf community coll ege contracts.
My staff and I will be pleased to discuss or clarify items in the report.
Respectfully submitted,
Dougk& R. Morton
Auditor General
Staff: Will iam Thomson
Peter N. Francis
Barbara Nienstedt
Nancy Lcvell
Cindy G. Whi taker
Gregory J. Halemba
Lucinda A. Trimble
Enclosure
2700 NORTH CENTRAL AVE. SUITE 700 PHOENIX, ARIZONA 85004 ( 602) 255- 4385
SUMPIARY
The Office of t h e A u d i t o r Generzl has conducted a performance a u d i t o f the
Arizona Department o f Corrections ( DOC ) contracts management function.
This audit, one i n a series on DOC, was conducted i n response t o a J u l y
26, 1985, resol u t i o n of the J o i n t L e g i s l a t i v e Oversight Committee.
The Department of > rrections Contracts Administration O f f i c e develops
approximately 300 -. ofessional and outside services contracts annual 1 y,
c u r r e n t l y valued a t more than $ 12 m i l l ion. The Office i s responsible f o r
procuring professional services and devel oping contracts, maintaining
contract documentation, and a c t i n g as a l i a i s o n between Department program
areas and contractors. Contracted services include purchase o f care
programs, hospital and medical services, counsel i ng services, educational
programs, etc. The findings i n t h i s r e p o r t r e f l e c t problems with c u r r e n t
and p r i o r year contracts.
DOC Needs To Strengthen I t s
Contract Devel opment And Sel ection
Process To Ensure F a i r And Open
Competition ( see , pages 5 through 13)
Currently, the Department's contracting process does n o t ensure f a i r and
open competition. DOC f a i l s t o follow the State Procurement Code i n
selecting contractors. A random saniple o f 40 contracts indicated t h a t i n
81 percent o f the requests f o r proposals ( RFP) r e q u i r i n g w r i t t e n
evaluations, DOC d i d n o t perform an evaluation ( 45 percent) or could not
provi de documentation o f the eval uation ( 36 percent). I n aadi t i o n , even
when DOC d i d perform eva? uations, it d i d not always use evaluation
f a c t o r s s t a t e d i n the RFP. Lack o f adequate proposal evaluations and
documentation could s u b j e c t t h e Department t o proposer protests and
1 awsui ts.
Fai 1 ure t o comply w i t h the Procurement Code appears t o resul t p a r t l y from
inadequate a u t h o r i t y and s t a f f i n g i n the Department's Contracts
Administration Office. Horiever, i n the i n i t i a l phase o f our a u d i t the
DOC Director a1 so expressed concern over the possible e f f e c t s of
pol i t i c a l considerations on some contracts. A1 thouyii we did not f i n d
evidence of direct political influence, we found that DOC staff believe
such influence exists. DOC staff identified seven contracts ( total ing
approximately $ 1.7 mil 1 ion ) i n which they perceived pol i tical infl uence is
a significant factor i n contracting w i t h these service providers. T h i s
perception has led to some contract irregularities. For example, two
providers and DOC staff appear to consider monies for these providers as
pass- ttlrough funds. In past years both providers were apparently
designated as DOC contract reci pients before the procurement process even
began, because they were l i s t e d by name i n DOC budget requests. Further,
DOC has a1 1 owed inappropriate provisions i n contracts with those
providers. For exampl e, both providers ' con tracts call ed for
prepayments. The DOC Director intervened after receiving a written
Attorney General ' s opinion i n May 1986 and has ended this practice. In
addition, one contractor provides services to non- DOC clients using
contract funds. DOC s t a f f s t a t e that they have l i t t l e control over
contracting certain services due to perceived pol i tical infl uence.
To help ensure f a i r and open competition i n the contracting process, we
recommend that the State Purchasing Office review the ~ epartment's
contracting process and take appropriate action.
The Department does not have a formal contract monitoriny system, and
consequently, cannot adequately verify the contractor service levels or
ensure contractor compliance w i t h contract service reporting
requirements. Our review of 42 percent of the purchase of care contract
f i l e s identified that none of these contractors met all the reportirig
requirements. Even when reports are submitted, DOC does not always use
them to verify that services are being provided. For example, reports of
one service provider indicated that only 17 percent of i t s staff time was
spent on services stipulated i n the contract, while 79 percent of i t s time
was charged to " other hours." However, DOC staff never followed up on
t h i s report to see whether the contracted services were actually being
provided.
In addition, DOC contracts do not provide an aaequate basis for effective
monitoring of costs and services. One service provider with two contracts
totaling $ 800,000 does not submit any of the requested budgets or
expenditure reports to the Department, nor a r e c o s t categories or service
costs specified i n the contract. Therefore, DOC is unable to determine
whether the cost of services are reasonable.
The Department Of Corrections Needs
To Evaluate The Effectiveness Of I t s
Contracted Programs ( see pages 21 through 36)
DOC does not know which of i t s contracted programs are effective.
Although the Department is required by the 1985- 86 appropriations b i l l to
evaluate service providers' performance, DOC ' s current efforts to evaluate
program effectiveness are 1 imi ted.
Our s t a f f conducted evaluations of two major DOC contracted juvenile
programs. The evaluation of the OK Community counseling services for
Adobe Mountain Juvenile Institution ( A! iiJ I ) residents showed that the
program had no effect on residents' institutional behavior or behavior i n
the f i r s t six months on parole. Poor service delivery and excessive s t a f f
turnover may have contributed to the ineffectiveness of the OK Community
programs a t ANJI. DOC needs to address these two factors, and should a1 so
eval uate other OK Communi ty counsel i ng programs.
The second evaluation showed that the Arizona Boys Ranch ( ABR)
Conservation Program has had a significant positive effect on program
graduates. More than 65 percent of the ABR graduates successfully
completed their f i r s t six months on parole, compared w i t h only 40 percent
of the comparison group of nonprogram par01 ees. S t i l l , further research
is needed to determine whether follow- up care would enhance the
effectiveness of the ABR Conservation Program.
To meet its legislative mandate to evaluate programs, DOC should create
two to three full- time equivalent ( FTE) positions to evaluate the
effectiveness of i t s contracted programs.
The Department Of Corrections Needs
To Improve Fiscal Oversight And
Control Over I t s Community Col 1 ege
Contracts ( see uases 37 throush 44)
The Department needs to strengthen financial oversight and control over
its community coll ege contracts. DOC contracts w i t h community col 1 ege
d i s t r i c t s t o provide inmate education services a t its various prison
s i t e s . DOC's fiscal year 1985- 86 contracts total $ 1,196,544. Currently,
DOC does not audit direct program expenditures. Nor does DOC verify
full - time student equivalent ( FTSE) counts and the resul ting State revenue
generated by its coll ege programs. Because FTSE counts directly affect
revenue for the DOC programs, DOC needs to ensure that its programs are
fully credited for their enrol lment. To improve financial oversight, DOC
should audit d i s t r i c t records to verify FTSE revenue and direct
expenditures.
In addition to weak fiscal oversight of i t ' s community college contracts,
DOC may be paying too much for inmate college education. Currently, most
d i s t r i c t s credit DOC progratns using the lowest possible FTSE rates. Using
more favorable FTSE rates could reduce DOC's contract costs by $ 61,000 to
$ 104,000. In addition, i n two contracts, overhead costs appear high -
approximately one- thi rd of total program costs.
TABLE OF CONTENTS
INTRODUCTION AND BACKGROUND. . . . . . . . . . . . . . . . . . . .
FINDING I: THE DEPARTMENT OF CORRECTIOtiS NEEDS TO
STRENGTHEN ITS CONTRACT DEVELOPMENT AND SELECTION
PROCESS TO ENSURE FAIR AND OPEN COMPETITION. . . . . . . . . .
Page
1
Failure To Comply With The Code
Has Weakened T' Selection Process . . . . . . . . . . . . . . 5
Contracts Administration Office Lacks
Authority To Ensure Compliance . . . . . . . . . . . . . . . . 11
Recommendations. . . . . . . . . . . . . . . . . . . . . . . . 13
FINDING 11: THE DEPARTMENT OF CORRECTIONS
DOES NOT ADEQUATELY MONITOR CONTRACTS. . . . . . . . . . . . . 15
Due To Poor Monitoring, DOC Cannot
I d e n t i f y Contract Problems . . . . . . . . . . . . . . . . . . 15
Current Monitoring E f f o r t I s Fragmented
And Procedures Are Inadequate. . . . . . . . . . . . . . . . . 18
Recommendations. . . . . . . . . . . . . . . . . . . . . . . . 20
FINDING 111: THE DEPARTMENT OF CORRECTIONS NEEDS TO
EVALUATE THE EFFECTIVENESS OF ITS CONTRACTED PROGRAMS. . . . . 21
DOC ' s Current Eva1 uation E f f o r t s Are Limi ted . . . . . . . . . 2 1
OK Community ' s Counsel ing
Program At AbUI I s I n e f f e c t i v e . . . . . . . . . . . . . . . . 22
The Arizona Boys Ranch Conservation
Program Has A Positive Effect. . . . . . . . . . . . . . . . . 31
DOC Needs To Eval uate The Effectiveness
O f Contracted Programs . . . . . . . . . . . . . . . . . . . . 34
Recommendations. . . . . . . . . . . . . . . . . . . . . . . . 35
FINDING I V : THE DEPARTMENT OF CCRRECTIONS
NEEDS TO IMPROVE FISCAL OVERSIGHT AND
CONTROL OVER ITS COFlMUNITY COLLEGE CONTRACTS . . . . . . . . . 37
Background . . . . . . . . . . . . . . . . . . . . . . . . . . 37
The Department Needs To Improve
Fiscal Oversight Control . . . . . . . . . . . . . . . . . . . 38
DOC May Be Able To Negotiate More
Favorable Contract Provisions. . . . . . . . . . . . . . . . . 39
Recommendations. . . . . . . . . . . . . . . . . . . . . . . . 44
TABLE OF CONTENTS
P a g e
OTHER PERTINENT INFORMATION . . . . . . . . . . . . . . . . . . . . 45
AREAS FOR FURTHER AUDIT WORK . . . . . . . . . . . . . . . . . . . 47
DEPARTMENT RESPONSE . . . . . . . . . . . . . . . . . . . . . . . . 51
APP END1 X
I . TECHNICAL REPORT
LIST OF TABLES
Page
TABLE 1 - Comparison Of Par01 e Performance Of Participants
And Non- Participants In The OK Community
Counseling Program A t AMJI. . . . . . . . . . . . . . . 2 7
TABLE 2 - T- Test Comparisons Of Average Incident
Reports Generated Per Month By AMJI
Counseling Program And Nonprogram Youth . . . . . . . . 2 8
TABLE 3 - T- Test Comparisons Of Average Educational
Progress Scores In Math And Reading Of
AMJI Counseling Program And Nonprogram
Youth ( Standardized By Month) . . . . . . . . . . . . . 29
TABLE 4 - Mean Number Of Documented Counseling
Services Provided To Cottage Alpha
Residents Per Week. . . . . . . . . . . . . . . . . . . 3 0
TABLE 5 - Comparison Of Parole Performance Of Arizona
Boys Ranch Conservation Program Graduates
And Youth Paroled Directly From AWI. . . . . . . . . . 33
TABLE 6 - Contract Year 1984- 85 Comparison Of Contract
And D i s t r i c t Average Estimated FTSE Revenues. . . . . . 4 1
TABLE 7 - Comparison Of Contract Year 1984- 85 Actual DOC
Program FTSE Revenues To Actual Proportional
F T S E R e v e n u e s . . . . . . . . . . . . . . . . . . . . . 4 2
LIST OF FIGURES
FIGURE 1 - Total Contract Dollars For Fiscal
Years 1982- 83 Through 1985- 86. . . . . . . . . . . . 2
FIGURE 2 - Contracts By Program Area
For Fiscal Year 1985- 86. . . . . . . . . . . . . . . 3
FIGURE 3 - Evaluation Code Compliance Sample Results
Contracts Executed For Fiscal Year 1985- 86 . . . . . 6
INTRODUCTICN AND BACKGROUND
The Office o f t h e A u d i t o r General has conducted a performance a u d i t o f the
Arizona Department o f Corrections ( DOC ) contracts management function.
This a u d i t was conducted i n response t o a J o i n t L e g i s l a t i v e Oversight
Committee r e s o l u t i o n o f J u l y 26, 1985, and i s one i n a series o f audits of
the Department.
The Department / la.> , he a u t h o r i t y t o c o n t r a c t f o r services, however, i n
doing so it must comply with the Arizona Procurement Code ( Arizona Revised
Statutes S41- 2501 ). The Department grants approximately 300 professional
and outside services contracts annually. These services include:
Hospital and medical services - physicians, dentists,
r a d i o l o g i s t s , prosthesis, medical f a c i l i t i e s , etc.
0 Purchase o f care programs - f o s t e r homes, group homes,
r e s i dential treatment, day care programs.
0 Educational programs - communi ty c o l l ege programs, vocational
education programs.
0 Counsel Sng services - p s y c h i a t r i s t s , substance abuse counsel ing,
r e h a b i l i t a t i v e services, etc.
e Consulting services.
Organization And Process
To administer t h e c o n t r a c t process, DOC has a Contracts Administration
O f f i c e w i t h i n the Administrative Services Division. This O f f i c e has been
overseeing the c o n t r a c t process f o r approximately three years. P r i o r t o
t h a t , contracts were administered through DOC'S Purchasing Office.
A1 though contracting i s administered by Contracts Adrninistration, the
process i s f a i r l y decentral ized. Contracts Administration i s responsible
for processing contracts, acting as a l i a i s o n between the contractors and
the program areas, and maintaining c o n t r a c t documentation. S t a f f i n
program areas are responsible f o r i n i t i a t i n g contracts, eval uating
proposals, selecting service providers, and monitoring the contracts once
they are in place. Contracts are ajso reviewed by the Attorney General's
Office.
Extent of Contracting
The following figures summarize DOC'S use of professional and outside
services contracts - ver the past four fiscal years. As shown in Figure 1,
the amount of fui . allocated for professional and outside services has
increased by 131 percent. The allocation for contracted services for
fiscal year 1985- 86 is $ 12.2 million.
FIGURE 1
Total C o n t r a c t Dollars
FISCAL YEARS 1982- 83 THROUGH 1985- 86
15 , I
1983 1984 1985 1986
Number i n ( ) = Number of contracts*
* The 1983 and 1984 contract counts are estimates. Dollar amounts,
however, are actual amounts.
Source: Compiled by Auditor General staff from DOC f i l e s
2
Figure 2 summarizes contracts by program area for fiscal year 1985- 86. As
i l l u s t r a t e d i n Figure 2, Juvenile Services and Adult Institutions have the
majority of professional and outside service con t r a c t do1 1 ars.
FIGURE 2
Contracts by Program Area
FOR FISCAL YEAR 1985- 86
5
4.7 1
HRD FPM CO CS HS JS
Program Area
A1 = Adul t Institutions HRD = Human Resources and Development
CO = Central Office HS = Health Services
CS = Community Services JS = Juvenile Services
FPM = Facil i t i e s , Planning and Maintenance
Source: Compiled by Auditor General s t a f f from DOC f i l e s
The Juvenile and Health Services program areas have the greatest number of
contracts. In addition, Juvenile Services has some of the highest dollar
contracts for purchase of care programs. Since fiscal year 1982- 83,
Juvenile Services and Health Services contracts have increased in dollar
amount by 138 percent and 103 percent, respectively. Total contracts have
increased in do1 1 ar amount by 131 percent.
AUDIT SCOPE AND PURPOSE
Our audit of DOC's contract management function concentrated on the
contracting process f o r professional and outside services. The audit
focused on contract selection, monitoring and evaluation. Our detailed
audit work deal t mainly with Adul t I n s t i t u t i o n s , Communi ty Services and
Juvenile Services contracts, and focused p r i m a r i l y on the 1985- 86 contract
year. However, problems i d e n t i f i e d i n t h i s report have occurred i n past
contract years.
I n addition t o documentation col lected a t DOC, we conducted on- si te v i s i t s
a t several contractor locations t o review t h e i r records. The audit report
presents findings and recommendations i n four areas.
e The adequacy of DOC's c o n t r a c t s e l e c t i o n process.
The adequacy o f DOC's contract monitoring function.
The adequacy of DOC's e f f o r t s to evaluate i t s contract programs.
( As a p a r t of t h i s finding, Auditor General s t a f f conducted an
evaluation o f the effectiveness o f two of DOC's contracted
programs. )
e The adequacy o f DOC's oversight and negotiation o f con~ munity
col 1 ege contracts.
Due t o time constraints, we were unable t o address a l l p o t e n t i a l issues
i d e n t i f i e d during our preliminary audit work. The section Areas for
Further Audit Work describes these p o t e n t i a l issues ( see page 47).
The Auditor General and s t a f f express appreciation t o the Director o f the
Department o f Corrections and h i s s t a f f f o r t h e i r cooperation and
assistance during the audit.
FINDING I
THE DEPARTMENT OF CORRECTICNS NEEDS TO STRENGTHEN ITS CONTRACT DEVELOPMENT
AND SELECTION PROCESS TO ENSURE FAIR AND OPEN COMPETITION
The Department o f Corrections ' ( DOC ) c u r r e n t c o n t r a c t i n g process does not
ensure f a i r and open competition. Noncompl iance with the Procurement Code
has undermined t % i n t e g r i t y o f the Department's contract selection
process. The De; tment's Contracts Administration Office needs to be
strengthened to enforce procurement code compliance.
Failure To Comply With The Code
Has Weakened The Selection Process
The Department has f a i l e d t o comply with the State Procurement Code i n i t s
proposal selection process. Procedures required by the Code have not been
consistently f o l 1 owed. Lack o f a systematic sel ection method contributes
t o noncompliance. The DOC Director has also expressed concern about the
possible e f f e c t s o f p o l i t i c a l considerations on contract selection.
Selection methods do not c o n s i s t e n t l y f o l l o w the Code - The Department has
f a i l e d to follow the State Procurement Code i n i t s contractor selection
methods. A. R. S. $ 41- 2534.6. requires t h a t contracts be awarded:
. . . t o the responsible offeror whose proposal i s determined i n
w r i t i n g to be the most advantageous to t h i s state taking i n t o
consideration the evaluation factors set f o r t h i n the request f o r
proposals. No other f a c t o r s o r c r i t e r i a may be used i n the
eval uation.
A random sample o f 40 contracts and corresponding requests f o r proposals
( RFP) shows t h a t i n 81 percent o f the cases r e q u i r i n g evaluations, DOC d i d
not perform w r i t t e n evaluations ( 45 percent) or cannot document t h a t
evaluations were done ( 36 percent). I n many cases, s t a f f i n the program
areas did not perform w r i t t e n evaluations because they received only one
proposal. However, even i n these instances, w r i t t e n evaluations should be
performed to document t h a t the provider meets stated requirements and
Departmental needs. Figure 3 presents the r e s u l t s o f our random sample.
FIGURE 3
EVALUATION CODE COMPLIANCE SAMPLE RESULTS
CONTRACTS EXECUTED FOR FISCAL YEAR 1985- 86
Source: Compiled by Auditor General staff based on a random sample of DOC
contracts executed for fiscal year 1985- 66
The Department does not always use the stated evaluation factors in
justifying i t s selection decisions. For example, one program area
developed a method for rating proposal s. However, the eval uation factors
stated in the RFP were not considered when this method was developed.
Consequently, not a1 1 stated factors were used in evaluating proposal s.
Without performing adequate eval uations and maintaining documents, the
Department i s subject to proposer protest and l a ~ s u i t s . Authoritative
1 i terature states that evaluations shoul d have both the appearance and
reality of being objective and f a i r . The contract f i l e should contain
sufficient information to show the method of evaluation and basis of
award, to reduce the 1 ikel ihood of 1 egal action.
* DOC personnel claim that in addition to the evaluations that were
documented i n their f i l e s ( 18 percent), another 36 percent were
performed b u t the documentation had been inadvertently destroyed.
Formal selection method could reduce noncompliance - To reduce
noncompliance, the Department needs to establish and follow a systematic
method of awarding contracts consistent w i t h the Procurement Code.
A1 though one program area has drafted internal management procedures, DOC
has few policies and procedures governing the contract process. As stated
e a r l i e r , our random sample review indicated that DOC does not consistently
evaluate RFPs. Moreover, program area s t a f f generally do not use a formal
and systematic method of evaluating RFPs,* thereby providing no basis for
justification statements. For example, prosram area s t a f f may use their
own methods of evaluation, develop their own forms for selecting a service
provider or use one of Contract Admini s t r a t i o n ' s forms. Currently, s t a f f
i n only one program area consistently use an evaluation method based on
stated evaluation factors.
Several other s t a t e s use systematic methods to rate proposals. For
example, Florida uses an RFP rating sheet that " must identify the most
important components of the RFP and contain measurable c r i t e r i a by which
proposals can be rated and compared." Most other s t a t e s ' rating forms
l i s t the rating c r i t e r i a and attach a possible score to each component.
With this method, evaluators are able to document and j u s t i f y t h e i r
sel ec ti on.
Political considerations - During the early phases of our audit the
Department Director requested that we examine possible problems w i t h
service provider contracts. Because some legislators are closely
af f i 1 i ated wi t h the Boards of some provi ders, he expressed concern about
the possible effects of pol i tical considerations on these contracts.
A1 though we did not find evidence of direct political influence, we found
s t a f f perceived such i nfl uence exists and acted accordingly.
In the case of two service providers, DOC s t a f f a t t r i b u t e f a i l u r e to
adhere to the Procurement Code to pol i tical considerations. These
providers who hold seven contracts total 1 ing approximately $ 1.7 nil 1 ion
* Contracts Admini s t r a t i on has devel opea a contract process guide t h a t
includes three different evaluation forms to use i n selecting fiscal
year 1986- 87 contracts. Program area s t a f f have the option of picking
which form they use or developing their own form.
have, i n the past, lobbied for funds directly a t the Legislature.
Staff be1 ieve these providers are pol i t i c a l l y influential and that the
Legislature intends that they should receive these monies. Staff to1 d us
they fear political consequences should they f a i l to award these
providers their contracts. However, these fears appear to be based on
past rather than current incidents. For example, staff i n the Department
indicate that under a t l e a s t one prior administration pressure was
exerted through the Director's Office to award a contract to one of the
two providers. In this case staff were dissatisfied w i t h the proposed
a c t i v i t i e s and provisions of the contract b u t were told the contract had
to be signed because the job of the ( former) director was i n jeopardy i f
the contract was not signed. However, the current Department Director
has told us he has never been pressured to award contracts to any
agency. He intends t h a t a l l department contracts be awarded i n
compliance w i t h law. Despite this, DOC staff appear to continue to hold
the belief that these providers must be treated differently.
The perception of pol i tical infl uence among s t a f f , whether currently
founded or not, has undermined the department's contract selection
process as i t relates to these two providers. The following discussions
involving instances from the current contracts show how this perception
has resul ted in code violatiorls and questionable decisions in awarding
and writing contracts.
Con tractor A
Contractor A has three contracts w i t h the Department for s l i g t ~ t l y less
than $ 1 million. DOC has contracted w i t h Contractor A since 1975. The
number of programs contracted for have expanded from one counseling
program in one f a c i l i t y to the current ten programs in six f a c i l i t i e s , a
residential treatment program and two outreach conimuni ty based programs.
e Although the Depart~ ent puts these programs out for competitive
b i d , the contractor and DOC staff appear to consider monies
funding Contractor A's programs as pass- through monies.
Contractor A stated that he lobbies the Legislature directly to
obtain funds for his programs. In his opinion, the Legislature
puts these monies in GOC ' s budget. However, the appropriations
report does not specify either as a 1 ine item or by footnote that
DOC must spend the monies to fund Contractor A's programs.
According to a Joint Legislative Budget Commi ttee analyst,
stipulating that monies go to Contractor A via the appropriations
bill would violate procurement statutes. If the cost of a
service exceeds the Code's maximum d o l l a r l i m i t s , i t must be put
out for bid. However, DOC s t a f f reported two instances in which
Contractor A has contacted DOC concerning " their" funds which
were i n the DOC budget. This was prior to DOC selecting any fund
recipients.
DOC bud, documents indicate that even before the contracting
process - gins Contractor A is designated as a contract
recipient. One program is l i s t e d i n DOC'S budget request as a
line item, stated by name w i t h a dollar amount attached. This
contrasts w i t h most services contracted for by the Department i n
which the budget request s t a t e s the general category, such as
psychological counseling or residential placements, with a do1 1 ar
amount requested. Through open cornretition DOC then selects
service providers to whom these monies are allocated.
o According to contracting experts, a t l e a s t two of Contractor A's
proposals should not have been considered uritil a1 1 RFP
requirements were met. For the 1985- 86 contracts, RFPs requested
that the provider submit program budget information and fees for
services. For two of the contracts, Contractor A did not submit
any budget information and i t requested a lump sum allocation for
each program. There was no documentation relating fees to
service that would support a lump sum request. Ivithout this
information, DOC should not have considered these proposals
because i t could not determine whether the cost of the program
was reasonabl e.
o In the past, DOC had agreed to inappropriate prepay~ ient
provisions w i t h Contractor A. Rather than billing DOC for
services rendered as i s the case w i t h most contracts, tile
provider receives quarterly up- front payments. After receiving a
written Attorney General's Opir~ ion in May 1986, the DOC Director
intervened and ended these up- front payments as of July 1, 1986.
Comment
o Personnel throughout the Department stated that many decisions
are made as a result of perceived political considerations.
Because of Contractor A's perceived pol i tical infl uence, staff
feel that they have l i t t l e control or ability to refuse
Contractor A's requests. Problems resul ting from these
perceptions extend into the monitoring of these contracts ( see
page 15).
Con tractor B
Contractor B consists of four organizations providing similar services. , @
According to one of the organization's Directors, i n 1979 two of the units
formed a Statewide network. They received Federal funding to establ ish
two additional units. When the Federal monies were no 1 onger available,
the State picked up the costs of the programs. Currently, DOC contracts
for these service- t a cost of approximately $ 740,000.
Similar to Contractor A, Contractor B is cited as a 1 ine item i n
DOC'S budget request and DOC s t a f f appear to consider monies
funding these programs as pass- through monies. However, the
appropriations report does not designate monies for these
services or for Contractor B.
e Considering the funds as pass- through has resulted i n
inappropriate contract provisions. The contracts a1 1 ow for
Contractor B to provide services to non- DOC clients using
contract funds. According to one DOC s t a f f person, Contractor B
maintains that originally the legislative intent was to fund
Contractor B to provide services for anyone meeting the c r i t e r i a
and not just DOC clients. Therefore, Contractor B was able to
get i t s contracts written with the provision that non- liOC clients
could be serviced with DOC contract funds. However, the Attorney
General ' s Office recently advised the Department that i t " may not
contract or provide or fund services for persons who have not
been committed to the Departr; ient." Staff also stated that they
have tried to verify l e g i s l a t i v e i n t e n t b u t have not founa
anything to confirm Contractor B's contention.
o In addition, Contractor B is considered an automatic contract
recipient, which prohibits f a i r and open competition. For
1985- 86 contracts, DOC received five proposals for these
services. Four proposers were the organizations constituting
Contractor B y and one was an outside party. Contractor B's total
proposed progracl costs exceeded the budget request allocation.
None of the proposals were evaluated, although the four
traditional proposers were awarded contracts and the fif t h
proposal was rejected. DOC s t a f f said that the proposals kiere
not evaluated and the f i f t h proposal was rejected ljecause they
knew there would be insufficient funds a f t e r the other four were
funded.
a Further, one DGC s t a f f member told us that one year the KFF was
written w i t h Contractor B's name on i t , until Contracts
Admini stration said this was i n violation of the procurement code.
Final ly, Contractor B a1 so received quarterly up- front payments
until January 1986.
Comment
8 Again, because of Contractor B ' s perceived pol i tical infl uence,
s t a f f feel that they have l i t t l e control or a b i l i t y to refuse the
Contractor's requests. In addition, s t a f f feel that pol i tical
considerations override Departmental concerns.
Contracts Administration Office Lacks
Authority To Ensure Compl i ance
The Contracts Administration Office i s weak and ineffective i n enforcing
compliance w i t h the Procurement Code. The Office lacks authority to
properly develop and oversee contracts. The State Purchasing Office may
need to a s s i s t the Department i n addressing problems w i t h i t s contracting
process.
Contracts Administration 1 acks authority - A1 thoucjh Contracts
Administration is responsible for ensuring that contracts comply w i t h
statutes and regulations, its authority is 1 imi ted i n contract decisions.
Program area s t a f f prepare most of the RFP, review and evaluate proposal s,
select the contractor, and prepare the contract justification form.
Program areas perform these important aspects of the procuremen t process
w i t h 1 i t t l e or no partici yation from the Contracts Administration Gffice.
Presently, Contracts Administration has limited authority to ensure that
program areas follow uniform procedures and comply w i t h the State
Procurement Code. For exampl e, a1 though Contracts Administration s t a f f
request program area s t a f f to submit written eval uations and acieqliate
justification statements, i t cannot r e j e c t a selection decision i f this
information i s not submitted or i s insufficient. Top- 1 eve1 rnanagement has
not given the Contracts Administrati on Office cl ear authority to override
improper program decisions. The Office has recognized the deficiencies i n
proposal eval uations and justification statements described earl i e r ( see
page 5). I t recently prepared a contract process guide and revised the
evaluation forms. I t also trained DOC personnel involved in the
evaluation process. However, i t contends that program area staff sti 11
have the option to develop and use their own methods and ~ O P ~ S .
Contracting standards require that Contracts Administration be invol ved in
contract development and selection as a check on program areas. According
to the State Purchasing Director and professional procurement l i t e r a t u r e ,
Contracts Administration s t a f f should directly participate i n most of the
procurement process. The State Purchasing Director recommends contracting
s t a f f involvement for several reasons. First, Contracts ~ d m i nsit ration ,
s t a f f have the expertise in contracting and famil i a r i t y w i t h procurement a
law that program area s t a f f may not have. Contracts Administration's
involvement can ensure that DOC complies with the State Procurement Code.
Second, direct participation by contracting personnel could reduce the
potential for bias. Bias may occur, or i t may appear that the process is
biased, i f those evaluating the proposal s ( program area personnel ) consist
largely of those w i t h prior or continuing involvement w i t h current
providers.
Designating contract managers is one method used by two other s t a t e s to
combine contracts administration and program area expertise and
involvement. A program area person i s assigned to oversee one or more
contracts from the RFP phase through the duration of the contract.
Contracts Administration personnel are a1 so invol ved in the contracting
process to ensure that appropriate procedures are followed. Contracts
Administration personnel also provide information regarding current
contract pol icy and preparation.
In addition to not having sufficient authority, Contracts Administration
may not have enough s t a f f to adequately participate i n and oversee the
contracting process. The Con tracts Administration Cffice has only two
contract administrators to oversee 276 contracts val ued a t $ 12.2 mil 1 ion.
As shown i n Figure 1 ( see page 2), the number of contracts and the do1 lar
amount expended on contracted services has increased 131 percent i n the
past four years. However, according to DOC s t a f f , there has been no
corresponding increase i n the Office's administrative personnel to oversee
these contracts. With its current s t a f f size, the Office cannot actively
participate i n selecting a l l contracts.
State Procurement O f f i c e involvement may be needed - The State Procure~~ ent
O f f i c e ( SPO) of the Department of Administration ( DOA) may need t o a s s i s t
DOC i n addressing problems with Procurement Code compliance. SPO suggests
two options.* F i r s t , SPO could provide s t a f f i n a consulting capacity.
It would a s s i s t DOC i n e s t a b l i s h i n g a systematic contracting process and
developing the Contracts Administration c a p a b i l i t y . Second, b u t less
preferable t o $ 7 would be t o have an oversight capacity i n DOC's
contracting procc . I n t h i s case, SPO would review DOC's contracting
actions to determine whether DOC has taken appropriate action and has
followed the Procurement Code.
If the Department i s unable t o ensure f a i r and open conlpeti tion, SPO
indicated t h a t DOA could r e s c i n d t h e Department's a u t h o r i t y to procure
services. Houever, DOA would have to determine the severity o f the
Department's noncompl iance before taking such action.
CONCLUSION
The Department o f Corrections has not c o n s i s t e n t l y f o l 1 owed procedures
required by the State Procurement Code i n selecting and awarding
contracts. The Departtilent neeas t o develop a systematic method c j f
selecting contractors. I n a d d i t i o n , t h e Contracts Administration O f f i c e
should be strengthenea t o ensure t h a t DOC complies with the Procure~ ent
Code.
RECObiMENDATI ONS
1. The Department should establish and f o l l o w a systematic rriethod o f
s e l e c t i n g and awarding contracts. P o l i c i e s and procedures should be
developed as guide1 ines f o r t l ~ o s e i t i v o l ved i n the contracting
process. The Director should communicate a clear pol i c y t h a t
pol i t i c a l considerations stroul d not influence c o n t r a c t decisions.
* According to the SPO Director, e i t h e r option would have to be on d
l i m i t e d or reimbursement method.
2. The Department should give Contracts Administration adequate authority
t o enforce compliance with the Procurement Code.
3. The Contracts Administration Office shoul d a c t i v e l y p a r t i c i p a t e i n the
selection process. This may necessitate an increase i n staff.
4. The State Purchasing Office should review the Department's contract
process and dc * mine whether further involvement i s necessary.
THE DEPARTMENT OF CORRECTIONS DOES NOT ADEQUATELY MONITOR CGiiTRACTS
The Department o f Corrections ( DOC shoul d imp1 ement a formal contract
monitoring system. DOC's e x i s t i n g system does not adequately v e r i f y
whether contracted services are being rendered or whether costs are
appropriate. I n F . ' it i o n , central oversight o f the moni t o r i n g function i s
lacking. To ensi a coordinated monitoring e f f o r t , DOC must plan and
imp1 ement proper moni t o r i n g procedures.
Contract monitoring i s important t o ensure service provider accountabil i ty
and compl iance w i t h s t a t e d terms and conditions o f the contract.
Monitoring i s the periodic review and documentation o f the c o n t r a c t o r ' s
progress i n f u l f i l l i n g t h e s t a t e d terms and conditions o f the contract.
It includes i d e n t i f i c a t i o n o f areas where c o r r e c t i v e action i s requirea
and follow- up t o ensure t h a t c o r r e c t i v e action i s successfully taken.
A1 though government agencies can del egate a governmental function through
a contract, they are s t i l l responsible f o r t h a t function. Effective
monitoring v e r i f i e s t h a t t h e f u n c t i o n i s performed.
Due To Poor Monitoring, DOC
Cannot I d e n t i f y Contract Probl ems
DOC's monitoring systern does not ensure t h a t contractors are providing
s a t i s f a c t o r y services and a t a reasonable cost. The Department has been
l a x i n enforcing r e p o r t i n g requirements and reviewing reports submitted.
Moreover, many contracts do not provide an adequate basis f o r e f f e c t i v e
monitoring o f costs.
C o n t r a c t o r ' s r e p o r t s a r e not . adequately reviewed - DOC does not ensure
t h a t contractors comply with service r e p o r t i n g requi remen ts. Thus,
service 1 eve1 s cannot be adequately v e r i f i e d . Auditor General s t a f f
reviewed 16 of 38 ( 42 percent) purchase o f care contract f i l e s and noted
t h a t none of the contractors met a l l the r e p o r t i n g requirements. Most
contracts s t i p u l a t e speci f i c a l l y what type o f information i s requi red of
the contractor, how frequently it i s t o be submitted, anci to whom the
information should be sent. However, the contractors often do not comply
with a1 1 r e p o r t i n g requirements. The fol1 owing examples ill u s t r a t e
contractors ' 1 ack o f compl iance.
o Eleven of 16 purchase o f care contract f i l e s reviewed contained
requirements f o r reports d e t a i l i n g s p e c i f i c dates on which youth
received treatment services and what those services entailed.
None o f the purchase o f care contract f i l e s reviewed contained
these reports. This requirement i s s p e c i f i c a l ly stated i n these
purchase ' care contracts. Without t h i s information, DOC cannot
v e r i f y t a l l youth received treatment services.
DOC contracts with a s e r v i c e p r o v i d e r f o r counseling i n i t s a d u l t
and j u v e n i l e i n s t i t u t i o n s . The c o n t r a c t r e q u i r e s t h e p r o v i d e r to
submit by c l i e n t name what services were offered and how
frequently. However, the monthly reports submi t t e d ~ y the
contractor do not meet t h i s requirement. Thus, DOC cannot v e r i f y
whether the counseling was provided.
In the case o f a t l e a s t one service provider, DOC personnel are aware o f
noncompl iance wi t h reporting requirements. However, they are ilesi t a n t to
take c o r r e c t i v e a c t i o n due t o what they consider the p o l i t i c a l nature o f
the contract ( see page 8). For example, one s t a f f person i d e n t i f i e d t n a t
one c o n t r a c t o r ' s r e p o r t s did not supply i n f o r m a t i o n r e q u i r e d by tlie
contract. However, the i n d i v i d u a l was unwil 1 ing t o con t a c t the provider
because she feared the perceived pol i t i c a l influence o f the contractor.
Even when reports are submitted, reviews are not adequate. Review of
selected contracts and reports indicates t h a t DOC does not always use
these reports t o v e r i f y t h a t services are being provided. The f o l l o w i n g
example ill ustrates weak review o f reports submitted.
a Reports of one n o n p r o f i t contractor i n d i c a t e t h a t it d l l u t t e a
only 17 percent o f i t s time for i n d i v i d u a l and group counseling
i n DOC programs. These are the services DOC s p e c i f i c a l l y
contracts f o r with t h i s provider. However, an Auditor General
review of the c o n t r a c t o r ' s r e p o r t s for one quarter of the f i s c a l
y e a r i n d i c a t e d t h a t t h e c o n t r a c t o r charged DOC f o r 79 pob rcent of
i t s time i n " other hours." A cursory review could quickly
i d e n t i f y these problems i n service del ivery.
Proper v e r i f i c a t i o n of services r e q u i r e s both d e t a i l e d review of reports
submitted and on- si t e monitoring v i s i t s . A1 though one program area i s
c u r r e n t l y devel oping an on- si te monitoring pro5rar; l to be implemented
during f i s c a l year 1986- 87, DOC's on- si t e monitoring has beer1 1 i m i ted.
According to DOC s t a f f , o t h e r d u t i e s have a higher p r i o r i t y than on- site
monitoring. Through on- si te monitoring, DOC could reconcile services
reported t o the Department with on- site records. If the documentation
does not substantiate the reports, the Department could withhold f u t u r e
payment or t h e p r o v i d e r could be required t o reimburse DOC.
Contracts lack b a s i s f o r moni t o r i n y - DOC contracts do not provide a basis
f o r e f f e c t i v e monitoring o f costs and services. A review o f selected
contracts indicates t h a t DOC may need t o review proposals more thoroughly
t o establ i s h accurate service costs and incorporate more s p e c i f i c
provisions i n i t s contracts. The f o l lowing examples ill ustrate DOC's need
t o analyze proposed costs f o r services.
8 One n o n p r o f i t s e r v i c e p r o v i d e r has tkio contracts with COC
t o t a l i n g $ 800,000. Although requested i n the RFP, t h e s e r v i c e
provider d i d n o t submit a budget j u s t i f y i n g i t s proposed costs
f o r i t s services. The contract with the agency also does not
specify any cost categories or service u n i t s . As a r e s u l t there
i s no basis f o r determining whether costs are reasonable. The
c o n t r a c t o r ' s budget was subniitted t o DGC only a f t e r Auuitor
General s t a f f requested t o review it. The budget indicates t h a t
approximately 39 percent and 42 percent o f the costs t o r the two
contracts are overhead expenses. According t o DOC s t a f f , a
substantial amount o f the con t r a c t o r ' s overheaa expenses sucrr as
space, equipment and suppl i e s are provided by the Department.
Therefore, the c o n t r a c t o r ' s overhead expenses t o r those contracts
appear t o be excessive.
An independent study performed f o r DOC inaicatea t t i a t costs
should be reviewed f o r some contracts before c o n t r a c t s a r e
signed. The purchase o f care contracts have corilparaule rates f o r
s i m i l a r services. However, i n three o f these Frograilis,
two- thirds of the meals and a substantial p o r t i o n o f tne a a i l y
supervision are not provided by the contractor. Consequently, it
appears t h a t i n those cases costs should be less. Tile scuuy
recommended t h a t these program costs be reviewed.
Contract languacje may a1 so be too vague i n some cases to provide an
adequate basis f o r monitoring. According t o independent consul tan t s for
DOC, the Department's c o n t r a c t s l a c k s p e c i f i c language defining the
services it desires. For example, a c o n t r a c t c a l l s f o r education
i n s t r u c t i o n b u t does n o t state the l e v e l or extent o f i n s t r u c t i o n . One
contractor may provide a t u t o r f o r one hour each week whereas another rnay
provide d a i l y formal i n s t r u c t i o n . However, both c o n t r a c t o r s would meet
the provisions of the contract. As a r e s u l t o f these discrepancies the
independent consultants recommended:
. . . t h a t the Department e s t a b l i s h a consistent set o f
terms and d e f i n i t i o n s . . . t o be used i n future
contracts. D e f i n i t i o n s should include measurable
c r i t e r i a f o r each service. An example might be t h a t
GED preparation woul d mean regul a r l y schedul ed c l asses
taught by a c e r t i f i e d teacher while GED t u t o r i n g could
be less structured and provided by a para- professional.
To address t h i s concern, the j u v e n i l e services program area has drafted a
l i s t o f service s p e c i f i c a t i o n s t o b e t t e r define services desired.
The c o n t r a c t document should be the source o f c r i t e r i a by which contractor
performance i s monitored. According t o a c o n t r a c t i n g a u t h o r i t y , t h e
monitoring function should be b u i l t d i r e c t l y i n t o the contract. Contracts
should contain work statements t o guide t h e monitoring function. A work
statement defines t h e s e r v i c e and u n i t s o f service; it specifies
standards, 1 icensure requirements, service goal s, objectives and tasks.
Generally, a work statement included i n the contract becomes the b a s i s f o r
monitoring.
Current Monitoring E f f o r t I s Fragmented
And Procedures Are Inadeauate
The Department's monitoring system lacks coordination anc; adequate
procedures. The c o n t r a c t monitoring function has no central oversi5k1t or
coordination. Furthermore, i n corr, parison t o other agencies, LOC does not
adequately plan f o r or u t i l i z e proper monitoring procedures.
DOC'S monitoring function i s fragmented. Each program area i s responsible
f o r monitoring i t s contracts and devising i t s own nionitoring procedures.
Seven program areas w i t h i n the Department regul a r l y u t i l i z e professional
and o u t s i d e s e r v i c e contracts. Each program area has several i n d i v i d u a l s
responsible f o r monitoring. Based on discussions with DOC o f f i c i a l s ,
approximately 61 DOC s t a f f people are involved i n the monitoring process.
However, no one i n d i v i d u a l has overall responsi b i 1 i ty f o r moni t o r i ng
contracts i n each o f the program areas.
Frequently, program monitoring i s inconsistent. This inconsistency i n
procedures i l l u s t r a t e s DOC'S lack of strong central coordination or
oversight over the monitoring function. To strengthen central
coordination and oversight, adequate procedures need to be established.
Procedures utilized by other agencies could serve as a model. For
example, the Federal Bureau of Prisons and the Arizona Department of
Economic Security assign specific individual s as contract managers w i t h
responsibility t r oversee contracts ( see page 12). In the Florida
Department of H ~ L . $ 1 and Rehabil i tative Services,* each contract manager
is assigned responsibility for one complex contract, or a group of less
compl ex contracts. This ensures that contracts are monitored throughout
the term of the contract.
Monitoring procedures should be developed a t the beginning of the contract
period. Florida ' s Department of Heal t h and Rehabi 1 i t a t i ve Service's
monitoring procedures recommend that the contract manager begin planning
monitoring a c t i v i t i e s upon assignment of a contract. The contract manager
should finalize the monitoring work plan w i t h i n one month of contract
execution. General ly, a monitoring work plan incl udes two basic
a c t i v i t i e s : thorough review of written reports and periodic on- si te
visi t. s, including a schedule for monitoring a c t i v i t i e s .
CONCLUSION
DOC needs to implement a formalized contract monitoring system. I t s
current monitoring system is not sufficient to ensure that services are
provided. In soriie cases, DOC contracts do not proviae a basis for
effective monitoring of costs and services. Furthermore, COC k smoni toring
function lacks central oversight. To ensure a coordinated monitoring
effort, DOC must plan and imp1 ement proper monitoring procedures.
* This systenl is recognized as a model by the National Institute of
Corrections.
RECOMblENDATI ONS
1. DOC should formal ize contract monitoring procedures to ensure adequate
and consistent monitoring e f f o r t s throughout the Department.
Procedures should be implemented a t the beginning o f each contract
period.
2. DOC should s p e c i f i c a l l y assign a s t a f f person i n each program area to
be responsib f o r contract monitoring i n conjunction with the
Contracts Adm; I s t r a t i o n Office. Persons assigned shoul d periodical ly
conduct monitoring a c t i v i t i e s t o ensure t h a t contractors are providing
the services as stipulated i n the contract. A c t i v i t i e s shoul d include
desk review, on- site v i s i t s and audits.
3. DOC contracts should include adequate cost provisions, work statements
defining services and the u n i t s o f services required so the Department
can e f f e c t i v e l y rnoni t o r contracts.
FINDING I11
THE DEPARTMENT OF CORRECTIONS NEEDS TO EVALUATE THE EFFECTIVENESS OF ITS
CONTRACTED PROGRAMS
The Department o f Corrections ( DOC) does not know which o f i t s contracted
programs are e f f e c t i v e and which are i n e f f e c t i v e . Current DOC e f f o r t s t o
evaluate program effectiveness are 1 ini ted. An Auditor General s t a f f
evaluation o f OK Community's counsel ing program a t Adobe Mountain Juvenile
I n s t i t u t i o n ( A M I ) indicates t h a t t h i s program does not have any
s t a t i s t i c a l l y s i g n i f i c a n t e f f e c t on program p a r t i c i p a n t s ' i n s t i t u t i o n a ' i
behavior or subsequent parole performance. However, an evaluation
conducted by Auditor General s t a f f indicates t h a t the Arizona Boys Ranch
( ABR) Conservation Program i s having a s t a t i s t i c a l l y s i g n i f i c a n t p o s i t i v e
e f f e c t on t h e p a r o l e performance o f program graduates. DOC needs t o
regul a r l y conduct simi 1 ar program evaluations.
Program evaluation i s a c r i t i c a l management t o o l t o assess whether a
s p e c i f i c program achieves desired resul t s . With t h i s information an
agency can make informed p o l i c y decisions regarding:
0 continuation or expansion o f a program;
0 deletion, a d d i t i o n or modification o f c e r t a i n program components;
0 acceptance o r r e j e c t i o n o f a program's approach; and
0 a1 1 ocation o f scarce resources among competing programs.
Auditor General s t a f f cor~ ducted eval uations o f two major programs
contracted by DOC t o demonstrate the value of such information i n making
pol i c y decisions. For a d e t a i l e d discussion of these program evaluations
see Appendix 1 .*
DOC'S Current Eva1 uation E f f o r t s Are Limited
DOC does n o t systematical ly eval uate the effectiveness o f i t s contracted
programs. The Department recognizes the need for more systematic and
comprehensive eval uations, b u t recent e f f o r t s have been i n s u f f i c i e n t .
* The technical r e p o r t contained i n Appendix 1 was reviewed Ly two
independent professional eval uators who concurred with i t s findings.
2 1
DOC has not attempted to systematically evaluate the effectiveness of its
contracted programs. In the past, the Department has assigned parole
officers famil i a r with specific contracted programs to semiannual ly
evaluate these programs to determine the quality of services being
provided. A DOC administrator stated that these evaluations are not very
sophisticated and are more of a " popularity contest" than a comprehensive
assessment of the qua1 ity of services being provided.
The Department recognizes the need for more systematic and comprehensive
eval uations of i ts contracted programs. A DOC superintendent acknowl edged
that he could not demonstrate that any program ( contracted or in- house) dt
his f a c i l i t y was more effective than any other program and that this
ref1 ected a deficiency i n the Department's program eval uation
capabil i t i e s . In hopes of addressing this deficiency, DOC recently
contracted w i t h outside consultants to eval uate 32 contracted programs to
determine whether DOC is being provided the services contracted for and to
assess what impact these services are having on program participants.
However, because of time constraints and fiscal 1 imitations, the scope of
the study was limited " to determining i f the services are being offered a t
a minimally acceptable level ." No assessment of the effectiveness of
these programs was undertaken.
Department s t a f f had unreal i s t i c expectations of the effort neeaed to
evaluate these service providers. DOC original ly planned to conduct the
32 evaluations w i t h i n two to two and one- half months and for a total cost
of $ 50,000. However, rigorous evaluations of program effectiveness take
more time and e f f o r t t h d n cursory e v a l u a t i ~ n sc o n a ~ c t ~ bdy f i e l a s t a f f ,
Auditor General s t a f f used approximately 1,000 employe? Eioirrs ( 1.25 FTE
positions for 5 months) to complete their evaluations of two corrtracted
programs. Both of these programs were included in DOC'S recent evaluation
e f f o r t s discussed above.
OK Community ' s Counsel i ng
Program A t AbU I Is Ineffective
Counseling provided by GK Community staff for AI~ JII's Cottage Alpha
residents has had no s t a t i s t i c a l l y significant impact on Cottage residents'
institutional behavior or subsequent parol e performance. Poor service
delivery and excessive staff turnover may have contributed to this
program's ineffectiveness. Further research is needed to eval uate whether
other OK Community programs are effective.
Program description - As part of its juvenile services contract, OK
Community offers counsel ing services to residents of AM 1 ' s Cottage
Alpha. T h i s con+ - act also requires OK Community to provide similar
counseling servic to selected juveniles a t Catalina Mountain Juvenile
Institution and the New Dawn Center for Girls, as well as referred
juveni 1 es on conditional re1 ease status ( parol e) i n Phoenix and Tucson.
The total amount of this contract for fiscal year 1985- 86 is $ 350,000.*
The OK Community counseling program a t APIJI's Cottage Alpha has been
operational since 1979. T h i s counsel i ng supplements DOC'S own
educational, work and recreational program provided to youth a t the
f a c i l i t y . A t the time of program inception, Cottage Alpha was reserved
primarily for violent, hard- core male delinquents - a population for which
this service provider claims special ized expertise. The ~ i a j o r i t y of' CiK
Community counselors are ex- offenders. However, because of f a c i l i t y
expansion and institutional overcrowding, Cottage Alpha i s no 1 onger
reserved primarily for this population of juvenile offenders.
The contract stipulates that OK Community assign three full- time staff
members to provide counsel ing services i n Cottage A1 pha. These counsel ors
are to work with a l l youth placed i n the Cottage, not just those wit11
violent, hard- core delinquent histories. The Cottage A1 pha supervi sor
estimates that the average daily population in the cottage i s from 30 to
38 residents.
j; OK Community also contracts with the Department to provide counseling
services a t three adult institutions and t o s t a f f an adult community
based counseling program a t a cost of $ 450,000 for fiscal year
1985- 86. Finally, the service provider operates a ten- bed residential
treatment f a c i l i t y for male youth offenders between the ages of 9 and
18. For fiscal year 1985- 86, DOC has block purchased all ten beds in
this f a c i l i t y a t a cost of $ 190,500.
A f t e r consul t a t i o n wi t h AN I and OK Community s t a f f and administrators,
Auditor General s t a f f proposed t o examine the e f f e c t o f OK Community's
counsel i n g program a t AMJ I on p a r t i c i p a n t s ' i n s t i t u t i o n a l behavior and
subsequent parole performance. Two populations o f Cottage Alpha ( program)
youth were compared w i t h two randomly selected groups o f APfiJI ( nonprogram)
youth who had s i m i l a r demographic and delinquent h i s t o r y backgrounds b u t
were assigned to treatment cottages other than Alpha. The f i r s t set o f
compari son groups o f program and nonprogram youth was comprised
e x c l u s i v e l y o f j~ _>.; iles committed to DOC because o f a j u v e n i l e court
p e t i t i o n a l l e g i n g a serious crime against person." The second set o f
comparison groups consisted o f youth representative o f the general
population ( i n c l u d i n g serious offenders) i n Cottage A1 pha and AFUI .**
Parole suspension and revocation data were used to measure a youth's
parole performance. Educational progress while a t AMJI and the number of
i n c i d e n t reports f o r assaul ti ve behavior or possession/ consumpti on o f
ill egal substances were used t o contrast Cottage A1 pha residents'
i n s t i t u t i o n a l progress and behavior with those o f youth i n the comparisori
group.
Counseling has no e f f e c t - P a r t i c i p a t i o n i n the OK Community counseling
program a t Adobe Mountain Juvenile I n s t i t u t i o n did not have a
s t a t i s t i c a l l y s i g n i f i c a n t e f f e c t on p a r t i c i p a n t s ' performance ciuring tile
f i r s t s i x months on par01 e ( see Tab1 e 1, page 27). Analysis o f the data
indicates t h a t program and nonprogram youth cannot be considered d i f f e r e n t
i n terms o f t h e i r parole performance. Any differences i n the parole
performance o f program and nonprograrn youth were not s t a t i s t i c a l l y
s i g n i f i c a n t a t the .05 level.***
* Peti ti on offenses i n t h i s category i n c l ude homicide, sexual assaul t,
aggravated assault, robbery, kidnapping, felony endangerment and
sexual abuse.
** Time frames f o r i n c l u s i o n i n the f i r s t set of comparison groups rias
January 1, 1983 through January 31, 1985. For the second set o f
comparison groups, time frames o f J u l y 1, 1583 through December 31,
1984, were used.
*** A p r o b a b i l i t y of .05 or smaller i s generally accepted i n the s o c i a l
sciences as the standard o f s t a t i s t i c a l significance. This .05
standard i s used i n a l l subsequent t e s t s o f s t a t i s t i c a l significance
discussed i n t h i s report.
If any discernible trend is evident, nonprogram youth, i n general,
performed better on parol e than program participants ( especial ly those
w i t h serious del inquency histories). The success rate a t the end of six
months on parole of general population nonprogram youth was 10.6 percent
higher than for general popul atiorl program participants ( 53.1 percent to
42.5 percent, respectively) .* Among youth w i t h serious del inquency
backgrounds, the difference i n parole success rates increased to 28.1,
percent ( 75.9 percent to 47.1 percent). However, these differences were
not s t a t i s t i c a l l y significant.
Data summarized i n Tables 2 and 3 indicate that the OK Community
counseling program a t AMJI also did not have a s t a t i s t i c a l l y significant
measurable effect on participants ' institutional behavior or educational
progress. Again, nonprogram youth general ly performed better on these
measures than youth receiving counsel ing services. On the average, the
comparison group generated fewer incident reports per month than program
participants in both sets of comparison groups ( see Table 2, Page 28).**
Furthermore, only i n the general population comparison groups did program
participants score higher i n an educational progress category ( reading)
than their nonprogram counterparts ( .09 grade 1 eve1 s per rnontr~
* Success i s defined as not having been sent back to AM1 ( parole
suspension or revocation), not having been placed i n a residential
treatment program because of poor parole performance, or not having
been placed on parole absconder status w i t h i n the f i r s t six months on
par01 e. Fai 1 ure was construed as having one ' s parol e rev~ keu wi tili n
the f i r s t six months because of new charges or technical violations.
Partial failure was defined as having had one's parole teliiporarily
suspended, being placed in a residential treatment program because of
poor parole performance, or having been placed on parole absconder
status within the f i r s t six months on parole.
** Incident report scores were stanaardized by dividing the number of
incident reports generated during their stay i n treatment cottages a t
AM1 by the number of months incarcerated in these cottages.
vs. .06 grade levels per month, respectively) ( see Table 3, page 29) .* In
the other three educational progress comparisons, nonprogram youth scored
higher. However, as in the analysis of parole performance, the
differences in average incident report and educational progress scores
( math and reading) of program and nonprogram youth were not considered
statistical ly significant.
Poor service delivery - Poor service del ivery may have contributed to the
ineffectiveness of the OK Community counseling program at AVdI. A review
of OK Community case files of program participants indicates that these
youth received only a limited amount of counseling services ( see Table 4,
page 30). Although the average daily population i n Cottage Alpha is
approximately 36 to 38 residents and three full- time OK Community
counsel ors are assigned to provide these services, OK Community staff
documentation indicated that program youth only received an average of
slightly more than one hour of counseling per week.
OK Community monthly summary reports indicate that i t s staff spend only a
small percentage of their time counsel ing youth. For the three months in
1984 for which data were available, OK Community counselors spent only 35
percent of their time providing individual an6 group counsel ing services
t o Cottage Alpha residents. Furthermore, during these three months OK
Community assigned only two full - time counselors to this program even
though the contract call ed for three full - time counsel or positions. For
the nine months in 1985 for which data were available ( Ciarch thro~ gn
December), the three counsel ors assigned t o the AivM I counsel i ng program
spent approximately 20 percent of their tixe providing inuividual and
group counseling services to Cottage Alpha residents.
* Upon incarceration at AbUI, new residents are given a series ? f
educational tests measuring the grade level at which they are
currently functioning. Just prior t o re1 ease, the AIkJ I educational
staff administer the same battery of educational tests. Educational
progress scores were generated on youth in our comparison groups by
measuring the difference between these test scores and dividing this
difference by the number of months they were incarcerated at AtvUI.
TABLE 1
CObPARISON OF PAROLE PERFORMANCE OF
PARTICIPANTS AND NON- PARTICIPANTS IIV THE
OK COMMUNITY COUNSELING PROGRAM AT AMJI
General Po~ ualt i on
Nonprogram Youth Program Youth
Percentage Number Percentage Number
Fai 1 ure 34.7 17 30.0 12
Partial Fail ure 12.2 6 27.5 11
Success 53.1 - 26 42.5 - 17
chi- square = 3.340 gamma = -. 074( l) p = . I88
Serious Offenders
Fail ure or
Partial Fai 1 ure
Success
Nonprogram Youth Program Youth
Percentase Number Percentaae Nurnber
chi- square = 2.753 gamma = -. 559( l) p = .097
Gamma i; leasures the strength and direction of the relationship
between two variables. The value of gamma may vary from 0 ( no
relationship) to + or - 1 ( perfect positive or negative
relationship, respectively). As a general rule i n social science
research, a gamma equal to or greater than + or - .300 i s considered
substantial enough to report as a moderate or stronger relationship.
However, even though a gamma may be considered substhntial, one
cannot infer that a relationship of similar strength could be
expected in the population from which our study samples were drawn,
unless the probability ( p) of this relationship occurring by chance
is equal to or less than the benchmark level of .05.
TABLE 2
T- TEST ( 1 ) COMPARISON OF AVERAGE
INCIDENT REPORTS GENERATED PER MONTH BY
AbIJ I COUNSEI- ING PROGRHCI AND NGNPROGRAb1 YOUTH
Program Youth Nonprogram Youth
Incidents/ Nuciber of Incidents/ Number of Si gni f i cance
Month Youth Month You; Level s
General Popul ation .33 3 6 .19 4 9 .342
Serious Offender .23 15 .14 28 .330
T- test analyses are conmonly used in social science research to determine whether the differences i n average
scores on a given variable for two different populations are s t a t i s t i c a l l y significant when independent random
samples have been drai~ ti frorii each poptilation. T- tests are particularly appropriate for small samples.
Math Progress
TABLE 3
T- TEST COMPARISONS OF AVERAGE EDUCATIONAL PROGRESS SCORES
IN MATH AND READING OF AM1 COUtJSELING PROGRALI
AND NONPROGRAM Y OUTW ( STANDARD I ZED BY MONTH )
General P o ~ ualt i o n
Program Youth
Grade Level s/ Number o f
Nonprogram Youth
Grade Level s/ Nurnber o f
Month Youth Nonth Youth
Reading Progress .09 3 3 .06 38
Math Progress
Serious Offenders
Program Youth
Grade Level s/ Number o f
Nonprogram Youth
Grade Level s/ Number o f
Month Youth Month Youth
Reading Progress .03 13 . ll 2 2
Si gni ficance
Level s
Si gni f i cance
Level s
TABLE 4
MEAN NUMBER OF DOCUMENTED COUNSELING
SERVICES PROVIDED TO COTTAGE ALPHA RESIDENTS PER WEEK
Serious Offenders General Popul a t i on
Service Type Contacts Per ~ e e k ( l ) Contacts Per Week ( 1 )
I n d i v i d u a l Counsel i n g .39 .40
Group Counseling .66 .53
Other Counsel i n g Contacts .14
Total Counsel i ng Contacts 1 .19
I n d i v i d u a l and group counseling sessions are approximately one hour
i n length. Other counseling contacts vary from 15 minutes t o one
hour.
Excessive s t a f f turnover - Excessive s t a f f turnover and prolonged s t a f f
vacancies may a1 so be negatively impacting the del i v e r y of counsel ing
services by OK Community s t a f f , as well as posing p o t e n t i a l s e c u r i t y
problems f o r DOC. The contract permits the s e r v i c e p r o v i d e r t o t r a n s f e r
s t a f f from i t s various programs as it deems necessary. This has resulted
i n a very unstable program environment. I n 1984 one o f three counseling
positions assigned t o the ACUI counseling program was vacant f o r a t l e a s t
nine months. Furthermore, the two f i l l ed positions experienced a t 1 east
four s t a f f changes during the year. This t r a n s l a t e s i n t o a minimum
turnover r a t e o f 200 percent i n 1984. The minimum turnover r a t e was again
200 percent i n 1985.
Interviews w i t h AI\[ UI a d m i n i s t r a t i v e s t a f f i n d i c a t e t h a t turnover among
Cottage Alpha counselors has been a continual problem. I n some instances
new counselors have begun working i n Cottage Alpha before they idere
interviewed or had a background check conducted by AM1 a d m i n i s t r a t i v e
staff. Not only does t h i s contribute to a h i g h l y unstable program
environment, b u t p o t e n t i a l security problems could a1 so arise.
Further research needed - Further research is needed to evaluate whether
other OK Community programs are effective. OK Communi ty operates simil ar
counsel ing programs a t two other juvenile and four adult institutions.
However, these programs are not necessarily identical. For example, OK
Community administrators stated that the Cottage Saguaro counsel iny
program a t Catalina Mountain Juvenile Institution is more structured, w i t h
greater emphasis placed on formal counseling services, than is the Cottage
A1 pha counterpart
However, before any further eval uations are conducted, the Department must
address questions regarding poor service delivery and excessive s t a f f
turnover a t other OK Community institutional program s i t e s . I t is
premature to conduct evaluations of program effectiveness when i t is
unclear i f the program is being imp1 emented properly. Proble~; ls re1 ating
to poor service delivery and excessive staff turnover were also found i n
Auditor General staff reviews of OK Community monthly summary reports for
other of their juvenile and adult institutional programs.
The Arizona Boys Ranch Conservation
Proqram Has A Positive Effect
The Arizona Boys Ranch Conservation Program has had a s t a t i s t i c a l ly
significant positive effect on program graduates. However, more research
is needed to determine whether an aftercare component would increase the
percentage of camp graduates who perform successful ly on par01 e.
Program description - The ASR Conservation Program is an intensive, h i ghly
structured ten- week program that attempts to i nsti 11 in participants
responsibil i ty, sel f- discipl ine and a positive work ethic. Prograrii
participants are referred directly from Adobe and Catalina Mountain
Juvenil e Institutions. The program has been operational since February
1984 and will cost the Department $ 888,000 during fiscal year 1985- 86. DOC
block purchases a l l 40 beds i n the program a t a cost of $ 1,850 a month per
bed. Five Conservation Program sessions are to be conducted by ABR d u r i n g
the current fiscal year.
The program i s divided i n t o three segments. I n i t i a l l y each resident takes
p a r t i n a one week o r i e n t a t i o n during which youth are f a m i l i a r i z e d w i t h
program r u l es, assessed by program s t a f f , and have i n d i v i d u a l i z e d
treatment plans formulated. After completing orientation, program
p a r t i c i p a n t s spend t h e n e x t nine weeks r o t a t i n g between f i e l d and campus
segments.
During the f i e l d ; egments, youth work on outdoor projects f o r various
county, State and i d e r a l agencies. Residents and s t a f f l i v e i n and work
out o f a mobile work camp t h a t can t r a v e l throughout the State. The
segments are one week long and residents p a r t i c i p a t e i n a minimum o f four
segments during t h e i r stay i n the program.
During the campus segments, program p a r t i c i p a n t s are invol ved i n both
community and campus work projects, vocational assessments, and GED
preparation as needed. Campus segments are also one week i n duration and
residents p a r t i c i p a t e i n a minimum o f f i v e segments.
A f t e r c o n s u l t a t i o n with AMJI and ABR Conservation Program s t a f f and
administrators, , Auditor General s t a f f proposed to compare tire subsequent
parole performance o f a s e l e c t group o f program p a r t i c i p a n t s with a
comparable group o f youth paroled from APUI who d i d n o t p a r t i c i p a t e i n the
program.
ABR Conservation Progran~ i s e f f e c t i v e - Data summarized i n Table 5
i n d i c a t e t h a t p a r t i c i p a t i o n i n the ABR Conservation Program had a
s t a t i s t i c a l l y s i g n i f i c a n t posi t. ive e f f e c t on p a r t i c i p a n t s ' subsequent
performance d u r i n g t h e f i r s t s i x monttls o f parole. Of the ABR graduates,
65.4 percent successful ly c o ~ i pelt ed t h e i r f i r s t s i x monttis on par07 e,
w h i l e o n l y 40 percent o f the comparison group of AbUI parolees did SO. A
strong measure o f association ( cjan~ rna = .448) between program p a r t i c i p a t i o n
and successful parole performance was generated t h a t i s considered
s t a t i s t i c a l l y s i g n i f i c a n t ( p = .025).
TABLE 5
CONPARISON OF PAROLE PERFORMANCE OF
ARIZONA BOYS RANCH CONSERVATION PROGMPi GRADUATES AND
YOUTH PAROLED DIRECTLY FROM AMJI
Fai 1 ure
AN I Par01 ees ABR Graduates
Percentage Number Percentage Number
44.0 22 21.2 11
P a r t i a1 Fai 1 ure 16.0 8 13.5 7
Success 40.0 - 20 65.4 34 7
100.0 - 5 0 100.0( 1) - 5 2
chi- square = 7.327 gamma = .448 p = .025
Discrepancy due t o rounding.
Further research needed - Further research i s needed t o determine whether
an aftercare component would enhance t h e e f f e c t i v e n e s s o f the ABR
Conservation Program. The D i r e c t o r o f the ABR Conservation Program
contends t h a t a m a j o r i t y o f the youth who complete the program are i n need
o f " strong aftercare support. "
With young men sucti as these, we can provide s t r u c t u r e
for 10 weeks and provide them w i t h many opportunities
t o learn. We can get thern t o the p o i n t where they are
w i l l i n g t o s t a r t making changes; b u t without continued
strong support a f t e r they leave they w i l l f i n d
it . . . d i f f i c u l t t o maintain t h e i r p o s i t i v e e f f o r t s .
A riaticinal study o f j u v e n i l e treatment programs released i n 1978 terids to
val idate the Program D i r e c t o r ' s concerns. This study concl udes t h a t the
a t t i t u d e s and s k i l l s acquired by youth i n programs s i m i l a r t o the one
conducted by the Arizona Boys Ranch do not necessarily t r a n s l a t e i n t o
s p e c i f i c ski1 1 s needed t o succeed i n t h e i r home environments without
expl i c i t f o l 1 ow- up.*
* See Dennis Rornig, Justice For Our Children: An Examination o f Juvenile
Delinquent Rehabil i t a t i o n Prograri~ s, Lexington Looks, 1978, Chapters 4
and 8.
DOC Needs To Evaluate The
Effectiveness Of Contracted Programs
DOC needs to systematically evaluate the effectiveness of i t s contracted
programs. DOC should consider a1 1 ocating permanent s t a f f positions to
regularly eval uate its contracted programs. Furthermore, the Department
should take steps to ensure that specific program goals and measurable
indicators of goal achievement are included i n contracts and that data on
these indicators are consistently coll ected a t regular interval s.
DOC should allocate FTE ~ o stio ns to conduct Droaram evaluations - To meet
a recent legislative mandate, DOC should consider creating FTE positions
that will be responsible for evaluating the effectiveness of its
contracted programs. A footnote attached to DOC'S 1985- 86 appropriations
report requires that the Department develop a " standardized evaluation
system" to measure the performance of its service providers. The report
further s t a t e s that DCC cannot disburse any funds for contracted services
after December 31, 1985, without having such an evaluation system in
pl ace.*
Based on the amount of time and personnel required for Auditor General
staff to conduct evaluations of the OK Community and Arizona Boys Ranch
programs discussed previously, we estimate that a minimurii of two to three
DOC program evaluation positions are needed. I t would not be unreasonable
for a program evaluation s t a f f of this size to Conduct eight to 12
comprehensive eval uations of program effectiveness per year. This woul d
a1 1 ow for the evaluation of approximately 30 contracted programs ( total ing
$ 5,080,000 during fiscal year 1985- 86)** over three years. Furthermore,
once a comprehensive evaluation system has been established and data dre
regularly collected on indicators of goal achievement, the number of
evaluations conducted yearly by DOC program evaluation staff should
increase by a sizeable margin.
* To meet the intent of the footnote, DOC personnel and legislative
staff have met to develop methods for evaluating contracted programs.
** T h i s figure i s basea on the DOC purchase of care, substance abuse
monitoring and major institutional counsel i n g contracts.
I n c r e a t i n g these positions, DOC should take steps to ensure t h a t these
positions are placed i n an appropriate organizational u n i t t h a t promotes
evaluation s t a f f autonomy and ready access to p o l i c y makers and
i n d i v i d u a l program managers. Furthermore, DOC may be abl e t o suppl emen t
i t s in- house eval uation capabil i ty with outside consul tants.
Development o f program goals and i n d i c a t o r s o f goal achievement - The
Department o f Corrections needs t o ensure t h a t s p e c i f i c program goals and
measurabl e i n d i c a t o r s o f goal achievement are s p e c i f i e d i n contracts and
t h a t data on these i n d i c a t o r s are c o l l e c t e d a t regular i n t e r v a l s.
Auditor General s t a f f evaluations o f OK Community counseling and the
Arizona Boys Ranch Conservation Program were hampered because programs
goals and i n d i c a t o r s of goal achievement were often too general and
subjective. Furthermore, i n instances i n which s p e c i f i c program goal s and
i n d i c a t o r s existed or could be developed, data on these i n d i c a t o r s were
often sparse or i n a format t h a t could not be subjected to s t a t i s t i c a l
analysis.
CONCLUSION
The Department o f Corrections needs t o r e g u l a r l y conduct in- house
eval uations o f t h e e f f e c t i v e n e s s o f i t s contracted programs. Outcome
evaluations conducted by Auditor General s t a f f i n d i c a t e t h a t the CK
Community counseling program i n Cottage Alpha i s not having any
s t a t i s t i c a l l y s i g n i f i c a n t e f f e c t on Cottage r e s i d e n t s ' i n s t i t u t i o n a l
behavior or subsequent par01 e performance. However, p a r t i c i p a t i o n i n the
Arizona Boys Ranch Conservation Program i s having a s t a t i s t i c a l ly
s i g n i f i c a n t p o s i t i v e e f f e c t on p a r t i c i p a n t s ' subsequent par01 e performance.
RECOMMENDATIONS
1. To meet i t s l e g i s l a t i v e mandate, DOC should create 2 to 3 FTE
positions t h a t w i l l be responsible for e v a l u a t i n g t h e e f f e c t i v e n e s s of
i t s contracted programs
2. DOC needs to take steps to ensure that specific program goals and
indicators of goal achievement are clearly specified i n contracts and
that data on these indicators are collected a t regular intervals as
part of the contract monitoring process.
3. Based on the two evaluations conducted by Auditor General s t a f f , DOC
shoul d:
a. take steps to address questions concerning poor service delivery
and excessive staff turnover i n the Cottage A1 pha counsel i n g
program and other OK Community institutional programs,
b. conduct program evaluations of all OK Community counseling
programs i n adul t and juvenile institutions, and
c. determine what effect the addition of an aftercare component to
the ABR Conservation Program would have on program participants'
subsequent parole performance.
FINDING IV
THE DEPARTMENT OF CORRECTIONS NEEDS TO IMPROVE FISCAL OVERSIGHT AND
CONTROL OVER ITS COtIMUNITY COLLEGE CGNTRACTS
The Department of Corrections ( D O C ) needs to improve the monitoring of its
community coll ege contracts. Because fiscal oversight and control over
these contracts i c weak, DOC may be paying more than is necessary. In
addition, the Dep- : lent may be able to negotiate more favorable college
contract provisions.
Background
The Department of Corrections con t r a c t s through intergovernmental
agreements with community college d i s t r i c t s to provide inmate education
services a t various prison s i t e s . DOC clrrrently has contracts w i t h Pima,
Graham, Maricopa and Pinal County Community College Districts and is i n
the process of contracting with Cochise County Conimuni ty Col lege
District. The d i s t r i c t s offer academic and vocational college credit
courses to inmates. As of March 1986, DOC'S fiscal year 1985- 86 col le5e
contracts total ed $ 1 ,196,544.
Qua1 ified community col leye d i s t r i c t s receive State funds for operatiny
expenses, based on full - time equival ent student enrol lments ( FTSE ) .* The
State pays the d i s t r i c t s a t various rates, depending upon the nurriber of
FTSE generated. ** The intergovernmental agreements require DOC to pay al l
program costs, including tuition and fees, less the FTSE monies received
for DOC students.
* FTSE counts are used in determining d i s t r i c t State aid. Different
State aid categories exist. For example, d i s t r i c t s receive basic
State aid for all academic courses b u t can also receive additional
funds for vocational or technical courses.
** In fiscal year 1584- 85, two rates were used for both academic and
vocational FTSE reimbursement, and three rates were used for each
category in fiscal year 1985- 86. FTSE count t o t a l s determine whicil
rates d i s t r i c t s will be entitled to use for reimbursement purposes.
The De~ artrnent Needs To Improve
Fi scal Oversight Control
DOC needs to strengthen contract financial oversi ght control s. Currently,
DOC does not verify the accuracy of program direct expenditures. The
Department does n o t verify d i s t r i c t FTSE counts and resulting program
revenues. In addition, one program i s currently carrying excessive cash
balances. By contrast, other State agencies have internal auditors to
verify program fin; - ial activity.
Program expenditures not audited - The Department has not recently audited
direct program expenditures. DOC receives quarterly expenditure program
summary reports from the districts. However, this information does not
verify whether individual expenditures within report categories were
proper and were made for DOC programs. In addition, unusual expenditures
may not be revealed without an audit of d i s t r i c t year- end records.
Although some DOC staff involved in the negotiation and contracting
process believe that d i s t r i c t records of DOC programs should be audited,
the Department has n o t initiated any recent audits. In contract year
1984- 85, for example, DOC was concerned that a district had changed
expenditure categories and a1 1 owabl e amoui~ ts within those categories,
without DOC authorization. A1 though ini tial ly concerned about possible
contract violations, DOC eventual ly re1 ied on the d i s t r i c t ' s correction of
these problems without any independent review or verification.
Excessive cash b~ lance exists a t one d i s t r i c t - DOC has allowed excessive
cash balances t o accumulate for one college program. Cash balances are
excess program monies and occur when revenues exceed expenditures. For
example, more FTSE revenue or fewer expenditures might have occurred than
originally projected in the contract budget. In this one instance, DOC
allowed a cash balance of $ 124,000 t o accumulate. This resulted in DOC
paying more than necessary for district services. Although DOC has since
taken action to reduce the cash balance, $ 56,900 s t i l l remains unused and
he1 d by the college.
State aid counts and revenues not verified - The Department does not
verify FTSE counts and resulting State aid revenue of its college
programs. The d i s t r i c t s include State FTSE money as revenue for the DOC
programs. DOC pays a1 1 program costs and expenditures not paid by the
FTSE revenue. Therefore, because FTSE counts directly affect revenue for
the DOC programs, the Department needs t o a u d i t pertinent coll ege records
to ensure that i t i s being credited a l l the FTSE revenue generated by i t s
programs. Although each d i s t r i c t ' s FTSE is audited each year by the
Audi tor General ' s Financial Audi t Division, these audits cannot assure DOC
that FTSE generated by its students i s properly credited to Department
programs. These are district- wide audits and are not designed to
determine FTSE and financial activity specific to DOC programs.
Other State agencies audit contractor records - Other Arizona agencies
have an internal audit s t a f f to ensure contract compliance and verify
program expenditures. The Department of Economic Security and the
Department of Health Services periodically audit contractor records to
ensure that program funds are properly spent. These two internal audit
groups range i n size from 12 to 15 members.
In contrast, the Department's three internal auditors do not audit
contractor records. According to Department personnel, t h i s i s because
the group has a small s t a f f and other assignments take precedence over
professional service contract audits. The audit group performs
operatiorial and programmatic f a c i l i t i e s reviews and is currently auditing
Department rev01 ving funds and institution cl ub funds. However, the
Department's Chief Internal Auditor be1 ieves that DOC should regularly
audit contractor records.
DOC May Be Able To Negotiate
More Favorable Contract Provisions
The Department may be able to negotiate more favorable contract provisions
with the community colleges. DOC may be paying more than necessary i n its
current inmate coll ege education contracts. In addition, DOC needs to
strengthen its review of completed contracts.
DOC may be paying more than necessary - The Department may be paying more
than necessary for inmate college education. DOC could reduce its
education contract costs by negotiating more favorable FTSE and overhead
cost contract provisions.
Currently, most d i s t r i c t s c r e d i t DOC program revenues using the lowest
possible FTSE rates. FTSE revenues decline as enrollment reaches
specified 1 eve1 s. In devel oping revenue projections, three of the four
d i s t r i c t s we reviewed credit DOC students as i f they were the l a s t to
enrol 1. * T h i s i s because they consider DOC programs as marginal or add
on. However, analysis indicates that DOC students currently represent an
increasingly 1 arge base of these col 1 eges ' enrollments. Currently,
Department programs account for nearly 20 percent of total FTSE counts for
some d i s t r i c t s . Further, actual total d i s t r i c t FTSE counts decreased a t
a1 1 the contracted d i s t r i c t s between fiscal years 1983- 84 and 1984- 85.
However, DOC contracted FTSE counts increased or remained stab1 e during
fiscal years 1984- 85 and 1985- 86. Therefore, DOC generated FTSE are
becoming an increasingly larger share of the d i s t r i c t FTSE.
The Department might have reduced i t s contract contribution by thousands
of dollars if i t had negotiated w i t h the d i s t r i c t s for higher FTSE
reimbursement rates for i ts programs. For exampl e, contracting for the
use of d i s t r i c t average FTSE rates woula have increased FTSE revenue for
the DOC programs, which would have reduced DOC'S cost for the programs.
Average rates can be calculated by dividing total expected d i s t r i c t basic
State aid revenue and vocational revenue categories by total projected
FTSE i n those categories.
* The use of these lower r a t e s r e s u l t s i n the least possible revenue
reimbursement to DOC. Since FTSE revenue and DOC contributions are
the only revenue sources for the DOC programs, the Department pays the
balance of program costs not credited as FTSE revenue. Therefore, DOC
pays more for i t s programs than may be necessary when d i s t r i c t s use
lower FTSE rates to credit Department programs. For example, i f a
d i s t r i c t reported 6,000 regular FTSE in fiscal year 1985- 86, the State
would pay the d i s t r i c t $ 1,030 per FTSE for the f i r s t 2,500 FTSE, $ 720
for 2,500 to 5,000 FTSE, and $ 510 for all FTSE i n excess of 5,000 ( in
t h i s case, 1,000 FTSE). Other rates apply to different State aid
categories.
Currently, one of the four d i s t r i c t s we reviewed uses projected average
rates for basic and vocational FTSE reimbursement. Table 6 shows DOC
potential savings of approximately $ 104,800 for contract year 1984- 85 when
average r a t e s a r e calculated and applied to the other three d i s t r i c t s .
Using t h i s method i n fiscal year 1385- 86 would have reduced DOC'S contract
costs by approximately $ 74,000.
TABLE 6
CONTRACT YEAR 1 984- 85 COMPARISON OF
CONTRACT AND DISTRICT AVERAGE ESTIMATED FTSE REVENUES
Additional DOC FTSE
DOC FTSE Revenue Contract Revenue
DOC Contract Using District When Average Rate
District FTSE Revenue Average FTSE Rates Is Used
Graham $ 283,244 $ 320,197 $ 36,953
Pi nal 253,720( 2) . 31 6,640 G2,22CI
Total Increase Of DOC Contract FTSE
Revenues With Use Of Average FTSE Rates
( 1) The calculations i n t h i s table were prepared using DOC contract
budget FTSE count projections, and total d i s t r i c t count projectiorls
obtained from the Arizona Coinnuni ty Col 1 ege Board.
( * ) The Pinal Coll ege D i s t r i c t c o n t r a c t ' s t o t a l vocational revenue i s
understated because of a math error. The correct vocational revenue
amount is used in t h i s Table.
Source: Prepared by Auditor General s t a f f from 1984- 85 DOC community
college contracts and d i s t r i c t budget information obtained from
the Arizona Community College Board
DOC might also have reduced i t s contract costs by negotiating for
proportional FTSE revenue reimbursement. As another FTSE reimbursement
option, proportional reimbursement i s determined by mu1 tiplying actual
d i s t r i c t S t a t e a i d received, or expected to be received, by DOC'S
percentage of actual FTSE counts to d i s t r i c t actual counts.* Using this
method, Department programs would be credited the amount of d i s t r i c t State
revenues actually generated by DOC FTSE.
As shown i n Table our analysis using the same three d i s t r i c t contracts
indicates that DOC could have saved $ 61,876 by negotiating proportional
FTSE reimbursement method.
TABLE 7
COMPARISON OF CONTRACT YEAR 1984- 85 ACTUAL DOC PROGRAM
FTSE REVENUES TO ACTUAL PROPORTIONAL FTSE REVENUES
Actual Proportional Additional
Program Reimbursement Of Proportional Revenue
D i s t r i c t Reimbursement Actual Revenue Over Actual Reimbursement
Graham $ 279,317 $ 37 7,929
Pima 84,780 90,949
Pi nal
Total Additional DOC FTSE Revenue Nhen
Proportional Reimbursement Is Used $ 61 ,876
Source: Prepared by Auditor General s t a f f from fiscal year 1984- 85 DOC
comrnuni ty coll ege contracts, Auditor General 1984- 85 Financi a1
Audit Division audit f i l e s , d i s t r i c t budget information obtained
from tile Arizona Coninuni ty Col lege Eoard, and information obtai rlcd
from the d i s t r i c t s
* t- or example, assume a d i s t r i c t generated 4,000 total FTSE during a
fiscal year and Department programs generated 400 FTSE, 10 percent of
the d i s t r i c t t o t a l . Based on fiscal year 1986 FTSE rates, actual
d i s t r i c t FTSE revenue would total $ 3,655,000, based on FTSE rates of
$ 1,030 for the f i r s t 2,500 FTSE and $ 720 for the remaining 1,500 FTSE
( 2,500 FTSE x $ 1,030) + ( 1,500 FTSE x $ 720). Therefore, proportional
reimbursement for the DOC procjram would be $ 365,500 ( 10 percent of
$ 3,655,000 ). Program revenue cal cul ated using the 1 owest rate woul d
total $ 288,000 ( 400 FTSE x $ 720), $ 77,500 less than proportional
revenue.
In addition to FTSE rates, the Department has not recently analyzed
overhead costs. Overhead costs charged to DOC programs total $ 498,500 for
fiscal year 1985- 66. Overhead categories incl ude general institutional ,
student safety and Office of the President. Although DOC i n the past
reviewed and eliminated broad account categories, i t has not analyzed
detailed expenses of remaining overhead accounts. DOC should examine
these costs to ensure that the prison programs receive indirect benefits.
DOC should t e s t District overhead expenditures for reasonableness and
propriety, and negotiate overhead r a t e s t h a t ref1 ect these benefits. This
is especially important because overhead accounts for nearly one- third of
total program costs i n two current contracts.
Inadequate review of completed contracts - The Department does not
adequately review the accuracy of completed community college contracts.
T h i s has resulted i n costly math errors, and inconsistent contract
provisions. Some State aid revenue budget cal cul ations were incorrect in
two recent contracts. In one contract, a calculation error combined w i t h
outdated FTSE rates increased DOC'S costs by nearly $ 10,700.
DOC ' s inadequate review of completed contracts creates other problems in
addition to overpayment. For example, a current contract is missing a
provision usually contained i n col 1 ege contracts. The provision
stipulates that the d i s t r i c t , i n conjunction w i t h DOC, will develop a
system of monitoring instructional qua1 i ty. D i s t r i c t s a r e to submit
monitoring results to DOC w i t h specified quarterly reports. The adequacy
of completed contract review i s especially important when, as in this
case, the d i s t r i c t prepared the contract rather than DOC.
Further, some contracts do not supply detailed FTSE count or program cost
and expenditure information. Even though most contracts contain some
detailed FTSE count and program expense information, the information does
not always agree w i t h other parts of the contract.
The Department needs to i~ iprove fiscal rnonitoring and preparation of i t ' s
inmate education community col 1 ege contracts. DOC needs to improve
contract financial oversight control. In addition, DOC may be able to
negotiate more favorabl e contract provisions.
RECOMMENDATIONS
1. The Department should:
a. Verify FTSE credited to its programs by the college d i s t r i c t s .
b. Revise current contract language to ensure current year contract
adjustment of cash balance carryover.
c. Review communi ty col 1 ege d i s t r i c t overhead account activity to
determine the appropriateness of the relationship of these
accounts to Department programs, and negotiate overhead rates
accordingly.
2. The Department should conduct audits of communi ty coll ege d i s t r i c t
year- end records to verify direct program expenditures. I t should
direct its internal audit s t a f f to perform financial audits and
reviews of relevant d i s t r i c t records. DOC may need to assess the
adequacy of the group's size.
3. DOC should pursue more favorable FTSE revenue contract provisions.
4. The Department needs to conduct thorough reviews of completed
contracts to ensure document accuracy.
OTHER PERTINENT INFORMATION
During our audit, we developed other information pertinent to one of the
Department of Corrections education contracts.
Inmates Received Grades For Canceled Courses
A legislator referred information to us alleging that an inmate had
received a grade for a course i n which he was enrolled b u t which had been
canceled. Our 1 imi ted foll ow- up shows that the col lege d i s t r i c t
inappropriately gave inmates grades and may have received State full- time
student equivalent ( FTSE) money.
Pinal Community College d i s t r i c t scheduled two courses for the spring 1984
semester, to be taught by the same instructor to Arizona State
Prison- Fl orence inmates. Original ly, the d i s t r i c t coded the courses as
prison courses b u t l a t e r recoded them as nonprison, main campus courses.
District personnel have confl icting opinions regarding the a1 1 eyed
cancel ed courses. The Registrar's Office contends that the courses were
not o f f i c i a l l y canceled. However, according to tire d i s t r i c t Is prison
campus dean and a course 1 isting, both courses were canceled during the
semester because the instructor fail ed to attend cl ass. [ vioreover, the
d i s t r i c t ' s Business Office canceled all the i n s t r u c t o r ' s contracts for the
semester after some payments had been made.
Inmate transcripts show that inmates received grades for both courses.
However, attendance documentation indicates that neither the 45- day FTSE
count class roster nor the final grade roster were signed by the
instructor. Rather, a Registrar Is Office s t a f f member signed the forms.
According to the ~ e g i s t r a'rs Office, i f students received grades State
FTSE money would have been requested.
\
This appears to have been an isolated occurrence. All credit hours
claimed for State Aid are susceptible to selection for testing in the
Auditor General financial audit of State Aid Enti t l erilents. The more
pervasive the occurrence rate of an error, the greater the likelihood that
erroneously claimed credit hours would be selected for testing and
detected. Such testing by the Auditor General ' s Financial Audit Division
has not disclosed additional instances.
AREAS FOR FURTHER AUDIT WORK
During the course of our audit, we identified potential issues that we
were unable to pursue due to time constraints.
Does DOC Adequately Detertnine When I t Is Appropriate To Contract For
Services?
DOC may not adequcA- ly assess when it is most appropriate to contract for
services rather t i provide them in- house. DOC contracts out for one
40- bed juvenile conservation program, a t a cost of $ 888,000 for the
current fiscal year. A similar 35- bed program, provided w i t h i n the
Department, is estimated by DOC t o c o s t $ 322,900 for fiscal year 1985- 87.
The Department did not perform any cost- benefit analyses i n deciding
whether to contract out for these services. Authoritative 1 i terature
suggests that contracting decisions be based on cost and quality
analyses. Further audit work is required to determine whether DOC
adequately decides when i t is beneficial to contract for professional
services.
Should DOC'S Community Based Service Contracts Provide Services To Non- COC
Cl i en ts?
Department documentation shovis that a t l e a s t two community basect service
organizations provide services to non- DOC clients as part of their DOC
contracts. The Attorney General s Office recently advised DOC that such
arrangements are improper. Further audit work is needed to determine the
extent of services financed by DOC for non- DOC c l i e n t s .
Is The Department's Contract Devel opment Period Sufficient Fur Timely
Contracting?
A sample of DOC contracts showed that 71 percent of the contracts were
executed after the date services were intended to begin. This occurs
\
because DOC does not beyin i t s contract process unti 1 April , which a1 1 ows
only three months for contract devel opment. When compl ications occur, the
process can extend beyond July 1, the effective date for most contracts.
Further, DOC often competes w i t h other government agencies for services
yet i t begins the contracting process two to three monttrs l a t e r than rnost
agencies. For exampl e, the Department of Economic Security begins the
process i n January, and places advertisements for proposals for services
beginning the f i r s t week of February. Authoritative 1 i terature suggests
that the process should begin six months before the expected service
starting date. Further audit work is needed to determine whether
competition has been impaired and to establish an adequate time period for
the Department's professional service contracting process.
1601 WEST JEFFERSON
PHOENIX. ARIZONA 85007
( 602) 255- 5536
BRUCE BABBITT
GOVERNOR
SAMUEL A. LEWIS
DIRECTOR
J u l y 2 , 1986
Douglas R. Norton
A u d i t o r General
O f f i c e Of The A u d i t o r General
2700 North C e n t r a l , S u i t e 700
Phoenix, Arizona 85004
D e a r M r . Norton,
The a t t a c h e d response t o t h e Department of C o r r e c t i o n s ' C o n t r a c t s
Management Function Performance Audit is provided f o r i n c l u s i o n
i n t h e t e x t of t h e published r e p o r t . This response has been
prepared as a r e s u l t of our review of t h e f i n a l r e v i s e d pre-l
i m i n a r y r e p o r t d r a f t which was submitted t o u s w i t h your
June 25, 1986 letter.
The Department agrees with your r e p o r t i n t h a t t h i s area
r e q u i r e s a t t e n t i o n . To t h a t end t h e Department has, i n t h e
p a s t twelve months, taken a c t i o n t o remedy t h e m a j o r i t y of
t h e i t e m s c i t e d i n t h e r e p o r t .
W e continue t o be d i s a p p o i n t e d with t h e a u d i t r e p o r t s published
on our Department. Your o f f i c e has not recongized t h e work t h e
Department has a l r e a d y undertaken o r accomplished nor d o e s y o u r
o f f i c e acknowledge t h a t t h e Department w a s aware of t h e problems
pointed o u t .
I f you have any q u e s t i o n s on t h e a t t a c h e d response, p l e a s e
c o n t a c t me at 255- 5497, o r Roger Austin at 255- 3525.
S i n c e r e l y ,
FINDING I
RECOMMENDATION / I1 - The Department should establish and follow a systematic
method of selecting and awarding contracts.
The Procurement Code was initiated in January 1985, six ( 6) months prior to the
beginning of a new fiscal year and approximately three ( 3) months prior to initiation of
the annual contracts procurement process for fiscal year 1985186. In t h i s time, the State
Purchasing Office had to train all Agency ~ u r c h a s i n g l C o n t r a c t ss taff regarding the new
code. Subsequently, c- lff training of line staff had to take place within each respective
S t a t e agency. In t _ short time, new forms were developed, training classes were
established, and some vrery basic training regarding the new code took place.
The procurement process for 1985/ 86 started with l i t t l e time to train and gain
familiarity with a totally new code which held staff far more accountable than ever
before.
Fair and open competition was encouraged and promoted at all times. Services were
advertised in accordance with Code requirements. Formal written evaluations did not
always occur, however, but this happened more from lack of understanding than from a
willful intent to violate the code.
During this time, there was not one protest filed by any offeror indicating that they f e l t
their proposal had not been fairly considered. All proposals received during this time
were available for review by the Auditor General's staff. Analysis could have been
performed to ascertain if, in fact, program staff had erred in selection of offerors whose
services were not advantageous to the S t a t e . Errors were admittedly made, however,
they were not made to the detriment of the S t a t e in the majority of the contracts
awarded. Those that remain questionable are those with " perceived" political pressure.
Fiscal 1985/ 86 was a stepping stone to identifying specific problem areas t h a t occurred
relative to the new Procurement Code. The Contracts training offered for 1986187
addressed those weak points. A great deal of time and effort was expended in developing
evaluation tools, with input from the S t a t e Procurement Office, to be used by the
Program areas. We attempted to make all the puzzle pieces of the Procurement Code f i t
together to promote better understanding of how all the component p a r t s r e l a t e to each
other.
Staff are now fully aware t h a t they are required to perform written evaluations of each
proposal and t h a t t h e evaluation factors relate to these identified in the RFP. All
evaluations, Justifications of Contract Awards and Requests f o r Contracts must be
signed by evaluators, supervisors and appropriate Assistant Directors before submission
to the Contracts Cffice. Staff understand that they must utilize the forrns provided by
the Contracts Administration Office unless they are able to identify another e f f e c t i v e
evaluation tool. If they e l e c t to use another tool, i t must be in the form of a written
evaluation and rnust include all factors identified in the RFP and must be forwarded for
inclusion in the contract file.
A Contract Guide was prepared and distributed in April to a l l program staff involved in
the contract process to ensure pertinent guidelines and processes are readily available
for review. Updates to Guides are provided as necessary.
RECOMMENDATION 82 - The Department should give Contracts Administration
adequate authority to enforce compliance with the Procurement Code.
The Contracts Administration Office staff fully understands t h e Procurement Code in
regard to the procurement of Professional Outside Services. The o f f i c e has established
uniform processes t o b e followed in order to ensure compliance with the Code.
Errors in processing a r e c o r r e c t e d and c o n t r a c t s are developed properly in accordance
with RFP guidelines and requirements. However, as in any organization, if a Director or
Assistant Director requires t h a t a specific contract be written with a particular clause
and the Attorney Ger - a1 does not preclude such a clause, the contracts are prepared as
directed.
It should be noted t h a t until 1985136, the Attorney General's Office had signed off on the
contracts aforementioned in the report without any indication of impropriety and, in
f a c t , f o r those c o n t r a c t s r e f e r r e d to on Page 10; had signed off with no objection t o
payment for non- DOC clients.
Additionally, the auditors seem intent on interpreti. ng a statement made during the
training conducted for the 1986187 c o n t r a c t y e a r to be one t h a t encourages violation of
the Code regarding RFP evaluation. The statement was o n e t h a t indicated that if
program staff were aware of other evaluation forms or processes t h a t would better the
system, to please share the information t o permit o t h e r s the benefit of knowing of other
methods for evaluation. It was also reiterated t h a t t h e evaluation process utilized must
coincide with criteria identified in the RFP.
The code only requires that written evaluations be based on c r i t e r i a r e f l e c t e d in the
RFP. I t does not require, nor does the S t a t e Purchasing Office require, that specific
instruments or processes be followed. What is the purpose in quashing innovativeness or
creativity? DOC i s NOT in violation by allowing other evaluation tools to be utilized as
long as they are in accordance with RFP requirements/ guidelines. The Contracts Office
has provided the direction to program staff and we have offered methods, but we have
not dictated which is t h e b e t t e r method. We enforce the Code by ensuring the RFP
guidelines are adhered to.
RECOKMENDP, TION # 3 - The Contracts Administration Office should actively
participate in the selection process. This rnay necessitate an increase in staff.
The Contracts Office has been allocated additional s t a f l for FY 1986187. Additional
staff will be requested in the FY 1987/ 55 Budget.
RECOM? UIENEATION 84 - The S t a t e Purchasing Office should review the Department's
contract process and determine whether further involvement is necessary.
The Department disagrees with this recornmendation. We have taken corrective action
in many areas of this process and are now in compliance with the Procurement Code.
The Department's s t a f f i s capable of solving problems t h a t may exist with t h e c u r r e n t
contracting process. There is no reason for the State Purchasing Office to be involved in
this area.
FINDING I1
RECOMMENDATION !'/ I - DOC should formalize contract monitoring procedures to
ensure adequate and consistent monitoring efforts throughout the Department.
Procedures should be implemented at the beginning of each c o n t r a c t period.
Steps are being taken to correct the problems identified in the area of Purchase of Care,
however, monitoring and evaluation processes t a k e t i m e t o implement.
Juvenile/ Community Services has, with its 1986/ 87 RFPs and Contracts, i n i t i a t e d a
classification of services and has developed service specifications t o b e used in
conjunction with the ' 3ssification. These documents were developed specifically for the
Purchase of Care pr. ams. The Adult Community Service area is also taking steps to
utilize a similar proL ' 5s. All program areas are recognizing the importance to more
specifically identify services and performance levels required of the contractors in order
to enable the program area to effectively monitor the contracts.
This is part of the learning curve associated with the continual growth of conctracted
services within the Department and the accountability required by the Code, the
Legislature and the Department itself. Many positive steps have been taken since
January 1985. The mistakes t h a t have been made have been corrected and areas have
been identified to be strengthened and established.
FINDING I11
RECOMMENDATION 1/ 1 - To meet its legislative mandate, DOC should c r e a t e 2 - 3 FTE
positions that will be responsible for evaluating the effectiveness of i t s contracted
programs.
The Department is meeting the legislative mandate of providing the required information
to the JLBC s t a f f as stipulated by the JLBC. T o d a t e far this has been statistical
information in terms of c o n t r a c t s f o r services within the Department and have been
limited to a few subject areas. The Department will be evaluating the services procured
in FY 1986/ 87 by consultants hired with purchase of care funds set aside f o r t h i s purpose.
Recommendations 1'/ 2 and # 3
The Department concurs with these recommendations.
FINDING IV
RECOMhlENDATION !'/ I, !/ 2, and / I3
The Department has concerns in this area as well.
Each year the FTSE rate is questioned, but generally the cornmunity college districts
consider DOC programs as marginal or add on. The Department has recognized that the
reimbursement rate should be higher for DOC programs. However, as the auditors f a i l to
acknowledge, negotiations involve two parties who must be willing to discuss
alternatives. The Districts are not willing to negotiate on this matter. They realize
that, 1) the Department cannot procure similar services elsewhere for less money, and 2)
if they wait long enough each year, the Department will become anxious about having
education programs available for the inmate populations. They stall until the discomfort
zone arrives for DOC. Implementation of these recommendations depends solely on the
cooperation of the community colleges.
cornmuniw
" WHERE YOU GO TO FEEL OK"
800 North 1st Street Phoenix, Ar~ zona8 5004 257- 1076
MR. DOUGLAS R. NORTON
AUDITOR GENERAL
2700 NORTH CENTRAL AVENUE
SUITE 700
PHOENIX, ARIZONA 85004
In response to your letter of June 25, 1986, we want to
clarify the following items which you listed as difficult to
discern ( numbered as yours on pages 2- 5 of your letter):
1. Since we have been hired as counselors rather than sta-tisticians
and since the terms " significant" and " positive" are
used with great frequency in therapy, we appreciate your agree-ment
that those terms should have been more clearly defined.
Futhermore, stri ed of the language of decision theory and of
concern with personal probabilities, all that a significant re-sult
implies is that one has observed something relatively un-likely
given the hypothetical situation, but relatively more
likely given some alternative situation. Everything else is a
matter of what one does with this information. Statistical sig-nificance
is a statement about the likelihood of the observed
result, nothing else. It does not guarantee that something im-portant
or even meaningful has been found.
2. We agree that evaluation is a critical management tool.
Because we s h a ~ e that belief, we evaluate our programs in weekly
program coordinators meetings, weekly staff meetings and with
consistant frequency we evaluate individual staff members ( i. e.
attatched forms).
3 . OK COMMUNITY strongly emphasizes the need for a multi-cultural
counseling staff not only because of differences in
value systems, but also because of the diversities in languages
and dialects including Black English, the various dialects of
Spanish as well as gang lingo. However, neither the P. M. L. A.
Writing Guide nor - F- o- r- m- - a- n- d- - S- t- y le by Campbell has yet to validate
the acceptance of split infinitives in writings concerning scien-tific
research methodology. Any linguist or writing teacher or
manuscript reviewer would agree that English usage is in a con-stant
state of flux. However, the appropriate style form for
scientific writing is usually one of the last areas to accept
linguistive change. This premise has been validated by some out-standing
linguists in the field such as Drs. Raven and Virginia
MacDavid, Dr. Alva Davis and Dr. Mackie Rlanton.
4. Please review your own report in which you stated that
the superintendents knew little of evaluation methodologies.
Page 24, Paragraph 1: Merely because of the same set of indi-cators
were used for both groups does not mean that those indi-cators
were valid. Validity is the degree to which an instru-ment
measures what it purports to measure. The primary purpose
of the analysis covariance is to provide an adjustment of the
results of an exp2riment for differences existing among subjects
before the start of the experiment. Scores on a control variable
are used to adjust for chance differences among treatment groups
and to reduce error of variance.
Two concepts frequently used by psychologists and other be-havoral
researchers are independent variable and dependent var-iable.
The independent variable is used in regard to the absence
of external stimulation whereas the dependent variable is defined
as the measured changes in subjects as indicated by their responses,
for example, frequency of avoidance responses. It should be noted
that the independent variable is only one sufficient condition
among many which can affect the phenomenon being studied. For in-stance,
maze learning will be affected to varying degrees by the
size of the maze as well as the amount of food reward that is given. •
The paralel here of course is the prison environment, i. e. Alpha
Cottage restriction prison priviles exceed those of other cottages.
This supposedly was due to the fact Alpha Cottage students were
violent offenders and the reward system under the control of the
Department of Corrections staff, i. e. early bedtime, late hours,
early discharge, lengthened incarceration, would be equivelent to
food reward.
a
All experiments require that certain conditions be kept con-stant
because these conditions may affect in some way the responses
being measured.
Variable complementation is the consistency of comparison and
similarities with independent and dependent variables as well as
an analysis of variance. This difference is evaluated by coap-aring
the variations within the samples to the variations be-tween
the samples.
Page 24, paragraphs 2 and 3: We disagree with the validity
of your selection of indicators of behavioral change in spite of
the fact that the Department of Corrections agrees with you. Our
approach to tro, lbled youth is far more holistic and longitudinal
in its assessment of success, although we rarely have the time
to design a research study because of our involvement with young
people. Since the term " partial ' failure" has limited quantitive
value, it is scienrifically inappropriate.
Page 27 and following: Many areas of your scientific method-ology
are of questionable validity and reliability. The Technical
Report does nothing but reinforce that opinion.
1. Limits of Coun- seli ng Time: It is our understanding that
the time of your study was January 31st 1983 to January 31st 1985,
and on the basis of that we are curious as to what Ghe Department
of Corrections officials currently at Cottage Alpha or currently
at Adobe Mountain Juvenile Institution would be familiar with
the OK COMMUNITY program during that entire period of time. From
January 1983 to January 1985 there has been a complete turnover of
staff not only at Cottage Alpha but all other cottages at Adobe,
consequently there are few if any line staff that would be know-ledgeable
of the OK COMMUNITY at that time.
What you again have failed to indicate is during the two year
period of time is that two major projects were initiated by the
Department of Corrections staff in Cottage Alpha.
Length of program guidelines: the intent being for Cottage
Alpha students, due to their violent crimes, to spend more time at
Cottage Alpha for those crimes.
Secondly, a CPO concept ( correctional Programs Officer) with
emphasis on Departmgnt of Corrections personnel being more directly
involved in treatment issues. The length of program guidelines was
never effectivedue to the increase of violent crimes during that
period of time. This increase forced the Department of Corrections
to be unab, le to function within their length of program guidelines
i. e. 8 months minimum, 1 year maximum. The increase of course cre-ated
a situation where there had to be a rapid turnover to provide
room for incoming students. This is one of the main reasons violent
students were integrated into other programs. This reduction in
time has a dual impact; we're talking about a mean or norm of ap-proximately
three and a half months: And although you speak of the
period of time you used to 3etermine success ( first six months) on
parole, you did not indicate what you determine as an adequate amount
of time in order to affect criminal behavior. So, are we to Selieve
it is your contention that fourteen to sixteen years of behavior,
lower socio- economic community, inferior education, racial stigma,
unemployment, are factors not to be considered? that you ignore.
these factors and ignore the three and a half month period of
time he is actually at Adobe? Are you suggesting that sufficient
time to impact the negative aspects of his environment and life-style?
You speak of failure as being violation of parole with-in
the first six months of release. What you don't speak of is
what period of time each individual youth was in the program pri-or
to that period of time and if the counseling hours provided to
each student was provided by OK COMMUNITY staff and Department of
Corrections staff. Are we to believe you can determine which had
the greator impact? Are you attempting what no other socio- scien-tists
have attempted? To predict behavior by dealing with only
the affect not the cause?
It was the request of the Department of Corrections that the
OK COMMUNITY staff reduce the hours on week- ends. If you would
have carried your research further you would have seen the OK COMM-UNITYS
original proposal and every proposal until 1983 indicating
an eight hour week- end Saturday and Sunday. The rationale for the
change was an " over kill". Department- of Corrections staff felt
the students should have some time for themselves and since Saturday
was really a visitation day and consequently chaotic. Sunday was
considered a day of rest and for preparing for the coming week. You
imply that it was only recent that the'counL increased tremendously
in Cottage Alpha. We beg to differ. The Minors Unit is a direct
result of the increase of violent crime incarceration and recidivism
by violent juveniles.
Scheduling has always and will always remain a problem. Not
due to any restraints on the Department of Corrections but to the
fact there are only twenty- four hours in a day and we think we spel-led
out a typical days schedule, " Begining with... classes begin at
8: 30 to 11: 30 a. m., lunch begins at 11: 30 to 12 p. m., school class
is from 12 o'clock until 3 p. m., dinner begins at 4: 40 p. m., recre-ation
begins at 5 p. m., ect..
2. Staff Turnover: It is the philisophical concept of OK COMMUNITY
to hire from a high risk population; ex- offenders, ex- drug addicts,
ex- prostitutes, ect., and to have former members of our internal
programs thereby providing a dual service to the State of Arizona.
We believe we not only affect juvenile ricidivism but adult recid-ivism.
Why wasn't an effort made to pursue the validity of this con-cept
by contacting former staff members and discerning as we advo-cate
that they have been integrated into the mainstream of society
in a productive manner.
3. Campus- wide Impact: Again we are concerned about the feed-back
you are getting from former or current Department of Correct-ions
staff. We seriously doubt the validity of your source of
information. Especially when you indicate a Department of Correct-ions
Administrator has given you information about what transpires
on the campus at Adobe Mountain Juvenile Institution. Our exper-ience
has been, and if you are familiar with prison environment,
few if anq. Administrators know what's going on on a daily basis.
Come on, let's be serious: Who determines a non- Alpha youth? In
this period of time in question ... what. are we speaking about? Total
recall? Again, get serious:
EhlF LOYE7YE -__ SUPEKVXSOR : -_ _-- -
E? ZLOYEE TITTE: Feriod Cover&.:
A. F7ORK OC. ECTZVES PldD RESPONSIBILITIES
FACTOR I. 1DPFTFELITY ON TI23 JOB PS P WHOLE
- - 1 I j I Obcer~ ntion- of rules ( conduct, policies). I
~=_(~ opee_& wiioshn supervisors* t I 1 1
Relations with feLlow workezs*
williagneso t o learn,
-- t. t-- i- t- u- d cs : eta££ trafp. 3- ng* I'
- iI2p- e - t- ldab J. SJ.:- p~
- X-- n ttistbve an$ resourcefulness*
Versat3- 1ity* --
i . 1
Effecti-~ er, essi n organizing self & tasks 1 I - I
I
- D. ecisfvencss* , l I i
i i I i
10. Leadership* 1 ) 1 j 1 I
11, General adjunzment to coxlitLons of work
12. Other _ C_ -- -
FP CTOR 2. QUf LXTY OF WOIZIK
13. Accmzcy and thoroughness of work includ- i! I 11 1 ' 1
recor--- d keen$=% and ~ 9pe. rw ork j i 1 1 1 !
Neatness, clarity and acceptability of
14. wc~ k. -.
15. Sougdriess of j ~ d ~ r n e n tasn d- dec- i.- sions
Effactiveness in ngeting and dealing with
' I I I I
16. p o-~, l e ( c3. ients & general public a t work -
7 Fffectivencss in presenting ideas / facts. I - -- ( I l l ! I
18. mfER I I !
f:? k. CYCXS DEFINED IN KEY ON PI?. GE 3
FACTOR 3. FRO~~ UCrXTG'fESS
19. Pnownt sf accept;- Te war!? produced* 1 1 ; ! I I
J ! I 1 1 8 20, M- ee ting t'~ 3~~~ deIa: d'lii~ nets. i~