OFFICE OF THE STATE INSPECTOR GENERAL

COMMONWEALTH OF VIRGINIA

FINAL AUDIT REPORT

Audit of VITA’s Service Level Agreement Management

Report No. 2023-PA-008

June 28, 2023

Prepared By: SysAudits.com LLC

June 28, 2023

The Honorable Glenn Youngkin

Governor of Virginia

P.O. Box 1475

Richmond, VA 23219

Dear Governor Youngkin,

The Office of the State Inspector General engaged SysAudits, LLC to conduct an audit of Virginia

Information Technologies Agency’s management of service level agreements. The report is included

below for your review and information.

OSIG would like to thank Virginia Information Technologies Agency Chief Information Officer

Robert Osmond and his staff for their cooperation and assistance during this audit.

Sincerely,

Michael C. Westfall, CPA

State Inspector General

cc: The Honorable Jeff Goettman, Chief of Staff to Governor Youngkin

Isabella Warwick, Deputy Chief of Staff to Governor Youngkin

The Honorable Margaret McDermid, Virginia Secretary of Administration

The Honorable Emily Brewer, House Chair, Communications, Technology, and Innovation

Committee

The Honorable Adam Ebbin, Senate Chair, General Laws, and Technology Committee

Robert Osmond, Chief Information Officer, Virginia Information Technologies Agency

Staci Henshaw, Auditor of Public Accounts

COMMONWEALTH OF VIR GI N IA

Office of the State Inspector General

Michael C. Westfall, CPA

State Inspector General

P.O. Box 1151

Richmond, Virginia 23218

Telephone 804-625-3255

Fax 804-786-2341

www.osig.virginia.gov

2023-PA-008

OFFICE OF THE STATE INSPECTOR GENERAL

Executive Summary

SysAudits, LLC performed the Virginia Information Technologies Agency audit on behalf of the

Office of the State Inspector General. The purpose of the audit was to assess and determine the

effectiveness of VITA’s management of service level agreements. Specifically, the audit objectives

were to determine:

1. If service levels are being met for the Supplier Strategy & VITA Performance contracts and

if there is accountability when service levels are not met; and

2. Whether the Service Level Management Program meets industry standards (NIST and ITIL)

for monitoring IT service contracts and provides effective tracking and monitoring as

prescribed in the contracts.

The scope of this audit included the Multisourcing Service Integrator (MSI) and its seven service

Tower support providers.

We conducted this audit in accordance with Government Accountability Office, Government

Auditing Standards. Those standards require that we plan and perform the audit to obtain sufficient,

appropriate evidence to provide a reasonable basis for our findings and conclusions based on our

audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings

and conclusions based on our audit objectives.

The audit resulted in identifying three opportunities to improve VITA’s service Tower monitoring

and nine recommendations. We determined that VITA’s oversight of monitoring the Tower metrics

was being performed in accordance with ITIL and NIST. However, the following are opportunities

to strengthen the delivery of Commonwealth of Virginia (COV) Tower contractor IT services:

Finding 1: While VITA's service level agreements (SLA) aligned with the National Institute of

Standards for Technology (NIST) 800-53 Service Acquisition (SA) and Information Technology

Infrastructure Library (ITIL) principles, VITA's SLAs required agency personnel to monitor a high

volume of metrics that did not add clear value to products and services. Specifically, VITA collects

and monitors 571 metrics for oversight of its Tower contractors on a monthly basis through critical

service levels, key measurements, and critical deliverables. VITA acknowledged that the 571-

service metrics may be too many and contain duplicative items that do not add value. Further, 94 of

the 571 metrics do not have a service credit mechanism to withhold funds from the contractor for

missing these service delivery requirements. The accountability for delivering the expected service

delivery is diminished without the monetary impact for which service credits can be imposed. From

our audit we made two recommendations:

1. Review Tower service levels that do not have a monetary impact and remove those that

add no value to COV agencies.

2. Develop a formalized process to evaluate SLAs for duplicative and inefficient metrics at

least annually and seek input from COV agencies in this process.

Finding 2: VITA did not share SLA metrics internally with agencies or formally categorize COV

Agency feedback on the Tower service providers. VITA’s current process includes obtaining

feedback through meetings (quarterly and monthly) with COV agency leads and service providers

2023-PA-008

OFFICE OF THE STATE INSPECTOR GENERAL

and by requesting agencies to respond to surveys on service provider performance. However, items

were not consistently completed. From our audit we made two recommendations:

1. Develop a process to formally compare customer feedback to SLAs on a periodic basis to

identify trends requiring SLA adjustments and those SLAs that were not met.

2. Develop a mechanism to formally share the expected metric requirements with agencies

that use service Tower support contractors.

Finding 3: VITA implemented processes to collect and review data needed to verify SLA

compliance, including a monthly review of contractor performance in meeting all SLAs. However,

VITA did not formally document those processes and the related methodology for completing its

review. From our audit we made five recommendations:

1. Evaluate the processes used to review the Monthly Validation Files to identify

automation where possible.

2. Evaluate the current contracts for the Service Tower Providers and determine if the

contract requirements and timelines of reviews should be modified to add additional

review time.

3. Review current resources for monitoring and managing contract requirements and

enhance resources and/or staffing as needed.

4. Determine if SLA Monthly Validation File sampling review is appropriate and document

the methodology that should be used if sampling is allowed.

5. Update the Monthly Validation File review process to formalize and standardize service

owner comments for unmet SLAs metrics that are submitted for exclusion (exclusion is

the process where contract vendors request not meeting the metric be excluded).

The Virginia Information Technologies Agency fully concurred with all audit recommendations.

David Cole, CPA, CISA, CRISC

SysAudits.com LLC

2023-PA-008

OFFICE OF THE STATE INSPECTOR GENERAL

Table of Contents

Page

Background

Objective

Audit Scope, Methodology, Sampling, and Criteria

Finding 1. VITA Service Level Agreements and ITIL Best Practices

Finding 2. VITA’s Processes for Managing Customer Feedback on

Service Provider Performance

Finding 3. VITA’s Monitoring of Service Level Agreements

Appendix – VITA Corrective Action Plan

2023-PA-008

OFFICE OF THE STATE INSPECTOR GENERAL

Background

VITA relies on the service providers listed below to ensure the Commonwealth has the IT

services required to meet its mission.

Services Provided

Platform

Suppliers

Contract #

EUS: End-User Services

Iron Bow

VA-180915-IBTL

MF: MainFrame

Peraton

VA-160926-

HPEN

MPS: Managed Print Services

Xerox

VA-191121-

XERX

MSG-NTT Messaging

NTT Data

VA-210517-NTT

MSI: Multisourcing Service Integrator

SAIC

TBD

MSS: Managed Security Services

ATOS

VA-180112-

ATOS

SSDC: Server Storage Data Center

Unisys

VA-180815-UC

VDN: Voice Data Network

Verizon

VA-151028-

MCI5

One of VITA’s key service agreements is the Multisourcing Service Integrator (MSI) and its

seven service Tower support providers. At the time of audit, there were 571 service metrics that

were included in service level agreements (SLAs) across the existing platform suppliers. The

suppliers are available to support VITA’s mission and enable the Commonwealth’s information

technology requirements. All suppliers that provide services to the Commonwealth are grounded

in using the Information Technology Infrastructure Library (ITIL) as a basis for providing

uninterrupted and high-quality service.

COV Tower Contractor Services Agency Survey and Feedback

The audit included a Tower service delivery survey to COV agencies. In addition, VITA

performs agency service delivery feedback surveys, as well as Joint Legislative Audit and

Review Commission (JLARC) who also has performed surveys. We compared the results of our

survey to the JLARC Survey and VITA Bi-Annual Survey from 2022 and determined that results

from all three surveys were similar.

2023-PA-008

OFFICE OF THE STATE INSPECTOR GENERAL

Audit Objectives

The objectives of this audit are to:

● Determine if service levels are being met by the Supplier Strategy & VITA Performance

(SSP) contracts. If service levels are not met, are vendors being held accountable?

● Determine whether Service Level Management Program meets industry standards for

monitoring information technology service contracts and provides effective tracking and

monitoring in the detail as prescribed in the contracts.

Audit Scope, Methodology, Sampling, and Criteria

Scope:

The audit scope included the Multisourcing Service Integrator (MSI) and its seven service Tower

support providers. The audit period of performance was November 2022 through May 2023.

Methodology:

The audit methodology included:

● Obtaining and reviewing the support service contracts and required SLAs.

● Identifying VITA contract monitors, and methods to monitor and document SLAs.

● Documenting deliverables and reporting requirements for each of the SLA vendors.

● Determining processes used to monitor and report SLAs to include documentation and

communication of SLAs not being met, and any resolution.

● Determine that VITA is evaluating compliance and using the tools built into the contract

to ensure compliance.

● Surveying impacted state agencies to review issues with service and delivery

requirements.

● Assessing SLAs to contract requirements to determine if SLAs align to contract

requirements, and are consistent with industry standards – NIST, ITIL, and other service

industries.

Sampling:

No sampling was performed. SysAudits.com LLC reviewed all metrics and SLAs outstanding

during the time of audit.

Criteria:

As part of the audit, several criteria were used to assess the audit objectives. The following

criteria were used to assess the audit objectives:

• Virginia’s Information Technology Resource Management, Information Security

Standard 501

This standard provides IT security policy and guidance for the Commonwealth of

Virginia.

2023-PA-008

OFFICE OF THE STATE INSPECTOR GENERAL

• NIST 800-53, External System Services (SA) 9

The following NIST guidance is recommended. Require that providers of external system

services comply with organizational security and privacy requirements and employ the

following controls:

1. Define and document organizational oversight and user roles and responsibilities

about external system services; and

2. Employ processes, methods, and techniques to monitor control compliance by

external service providers on an ongoing basis.

• ITIL Foundation 4

The following ITIL guidance was deemed applicable.

o Services is defined as: A means of enabling value co-creation by facilitating

outcomes that customers want to achieve, without the customer having to manage

specific costs and risks.

o Service offering: Is a formal description of one or more services, designed to address

the needs of a target consumer group. A service offering may include goods, access to

resources, and service actions.

o Service offerings may include:

• Goods to be supplied to a consumer (for example, a mobile phone). Goods are

supposed to be transferred from the provider to the consumer, with the consumer

taking the responsibility for their future use.

• Access to resources granted or licensed to a consumer under agreed terms and

conditions (for example, to the mobile network, or to the network storage). The

resources remain under the provider’s control and can be accessed by the

consumer only during the agreed service consumption period.

• Service actions performed to address a consumer’s needs (for example, user

support). These actions are performed by the service provider according to the

agreement with the consumer.

• ITIL Principles

o Do fewer things but do them better. Minimizing activities to include only those with

value for one or more stakeholders will allow more focus on the quality of those

actions.

o Respect the time of the people involved. A process that is too complicated and

bureaucratic is a poor use of the time of the people involved.

o Have processes that are easier to understand, more likely to adopt to embed a practice,

make sure it is easy to follow.

o Monitor: The governing body monitors the performance of the organization and its

practices, products, and services. The purpose of this is to ensure that performance is

in accordance with policies and direction.

2023-PA-008

OFFICE OF THE STATE INSPECTOR GENERAL

• ITIL Service Level Management

The purpose of the service level management practice is to set clear business-based

targets for service levels, and to ensure that delivery of services is assessed, monitored,

and managed against these targets:

o Definition: Service level. One or more metrics that define expected or achieved

service quality.

o Definition: Service level agreement. A documented agreement between a service

provider and a customer that identifies both services required and the expected service

level.

Service level management provides the end-to-end visibility of the organization’s services. To

achieve this, service level management:

o Establishes a shared view of the services and target service levels with customers.

o Ensures the organization meets the defined service levels through the collection, analysis,

storage, and reporting of the relevant metrics for the identified services.

o Performs service reviews to ensure that the current set of services continues to meet the needs

of the organization and its customers.

o Captures and reports on service issues, including performance against defined service levels.

The skills and competencies for service level management include relationship management,

business liaison, business analysis, and commercial/supplier management. The practice requires

pragmatic focus on the whole service and not simply its constituent parts; for example, simple

individual metrics (such as percentage system availability) should not be taken to represent the

whole service.

2023-PA-008

OFFICE OF THE STATE INSPECTOR GENERAL

Finding 1: VITA Service Level Agreements and ITIL Best Practices

While VITA's service level agreements (SLAs) aligned with NIST 800-53 Service Acquisition (SA)

and Information Technology Infrastructure Library (ITIL) principles, VITA's SLAs required agency

personnel to monitor a high volume of metrics that did not add clear value to products and services.

Specifically, VITA collects and monitors 571 metrics for oversight of its Tower contractors on a

monthly basis through critical service levels, key measurements, and critical deliverables. VITA

acknowledged that the 571 service metrics may be too many and contain duplicative items that do

not add value. Further, 94 of the 571 metrics do not have a service credit mechanism to withhold

funds from the contractor for missing these service delivery requirements. The accountability for

delivering the expected service delivery is diminished without the monetary impact for which

service credits can impose.

VITA recently started an initiative to review all SLAs to identify duplicative service level metrics

that do not add clear value to operations. Due to the high number of SLAs and its ongoing priorities,

VITA had not yet completed the initiative. Therefore, the SLAs that did not add clear value were not

yet removed or combined with other service level items.

Criteria: ITIL recommends for organization to “keep it simple and practical” for managing its value

stream.

Specifically, ITIL prescribes if a process, service, action, or metric fails to provide value or

produce a useful outcome, eliminate it. In a process or procedure, use the minimum number of steps

necessary to accomplish the objective(s). Always use outcome-based thinking to produce practical

solutions that deliver results. To apply the “keep it simple and practical” principle, ITIL

recommends organizations consider and perform the following to successfully manage the value

stream:

● Do fewer tasks but do them better: Minimizing activities to include only those with value for

one or more stakeholders will allow more focus on the quality of those actions.

● Respect the time of the people involved: A process that is too complicated and bureaucratic

is a poor use of the time of the people involved.

● Easier to understand, more likely to adopt: To embed a practice, make sure it is easy to

follow.

Duplicative SLA’s and SLA metrics without a monetary impact can hinder the COV in receiving

full benefits of contracted services.

Recommendations:

1. Review those Tower service levels that do not have a monetary impact and remove those that

are not deemed to add value to COV agencies. In addition, VITA should minimize Tower

service level requirements that do not include a monetary impact.

2. Develop a formalized process to evaluate SLAs for duplicative and inefficient metrics at

least annually and seek input from COV agencies in this process.

ITIL defines the value stream as an organization’s process for creating and delivering products and services to

its stakeholders.

2023-PA-008

OFFICE OF THE STATE INSPECTOR GENERAL

VITA’s Response:

1. VITA is removing over half of all SLAs and removing the category of SLAs that do not have

monetary impact with Project Evolution. This modification also is moving the program to a single

target for all SLAs, removing the need to measure each SLA 2 times, once for Target and once for

Minimum.

2a. VITA will develop a review cycle that will include input from the agencies through the Relationship

Management Committee (RMC). The effectiveness of this process will be dependent upon the

completion of the implementation of Project Evolution.

2b. Any recommendations for related changes based on this review will be reviewed with the Service

Owners and presented to the SLM Forum in one of the Monthly SLM Forum meetings for decision.

Estimated Completion Date: June 2024

2023-PA-008

OFFICE OF THE STATE INSPECTOR GENERAL

Finding 2: VITA’s Processes for Managing Customer Feedback on Service Provider

Performance

VITA did not share SLA metrics internally with agencies or formally categorize COV Agency

feedback on the Tower service providers. VITA’s current process includes obtaining feedback

through meetings (quarterly and monthly) with COV agency leads and service providers and by

requesting agencies to respond to surveys on service provider performance. However, items were

not consistently completed.

VITA periodically obtains data to understand agencies’ feedback on service provider support.

Additionally, VITA demonstrated instances of adapting to emerging service provider issues to

support agency concerns. However, VITA does not have a process to ensure that agency feedback is

tracked and categorically analyzed on a periodic basis to identify trends requiring further

investigation and SLA adjustments. An analysis of the last 12 months was unable to confirm a

process that demonstrates a consistent repeatable process used to periodically catalog, analyze and

compare customer feedback to SLAs to identify trends and adjustments that may be required. VITA

explained that it relied on other oversight mechanisms such as its governance forums with service

providers to address agency feedback and therefore implementing a process to periodically analyze

customer feedback and update SLAs was not yet completed.

Additionally, VITA did not have a process to ensure that agencies had an adequate understanding of

the metrics used to assess service providers. Specifically, VITA did not provide agencies with SLAs

or other guidance to assess and monitor service provider performance. Without the SLAs, agencies

explained they were uncertain about the specific metrics used to assess and monitor service provider

performance. For example, agencies explained they did not understand the methodology for

counting elapsed days for service tickets and the processes for determining whether agencies or the

service provider were responsible for service issues. VITA explained that it does not share SLAs

with agencies due to the high volume of SLAs and reliance on the Multisourcing Service

Integrator’s processes to facilitate discussions between relevant stakeholders on an ongoing basis.

Criteria: ITIL recommends organizations to have a process to collaborate and promote visibility.

Specifically, ITIL encourages organizations to work together across boundaries. This produces the

likelihood of greater buy-in, more relevance to objectives, and increased likelihood of long-term

success. ITIL also recommends feedback to be analyzed to identify improvement opportunities,

risks, and issues.

VITA’s lack of a formalized process to periodically analyze customer feedback for potential SLA

adjustments increases the risk of service issue trends not being identified and addressed timely.

Additionally, agencies do not have adequate visibility into the metrics used to assess service

providers; thereby, reducing their opportunity to provide meaningful feedback and identify larger

service issues. Overall, these risks pose additional risks to agency operations that rely on the Service

Tower support to complete their mission.

Recommendations:

1. Develop a process to formally compare customer feedback to SLAs on a periodic basis to

identify trends and adjustments that should be made to SLAs.

2023-PA-008

OFFICE OF THE STATE INSPECTOR GENERAL

2. Develop a mechanism to formally share the expected metric requirements with agencies that

use Service Tower support contractors.

VITA’s Response:

1a. VITA has developed a handoff process between the Customer Satisfaction Review Team and

the SLM Team.

1b. Research and determine the methodology. Then develop a work instruction for the SLM

Team that will be used to evaluate trends and adjustments based on the feedback received.

2. VITA will develop a mechanism that makes SLA definitions and metrics more easily

available for agency review. This is dependent on completion of Project Evolution and will

require resources outside the SLM team to implement.

Estimated Completion Date: May 2024

2023-PA-008

OFFICE OF THE STATE INSPECTOR GENERAL

Finding 3: VITA’s Monitoring of Service Level Agreements

VITA implemented processes to collect and review data needed to verify service level agreement

compliance, including a monthly review of contractor performance in meeting all SLAs. However,

VITA did not formally document those processes and the methodology for completing its review.

For example, we noted the following internal control weaknesses in the process:

● VITA service owners validate a Monthly Validation File (validation file) for each SLA by

sampling specific service tickets included in the file. However, VITA has not yet formally

documented the minimum standards for sampling or the criteria that service owners should

use. For example, the minimum number of items that must be sampled for review, formally

documenting items included in the sample, or factors justifying items that should be

prioritized for sampling (e.g., higher risk or visibility items) were not documented to ensure

that service owners had a standardized process for selecting samples. VITA explained that it

wanted service owners to have flexibility in reviewing large volumes of data and therefore

did not formalize a sampling standard. Further, VITA explained that service owners rely on

their knowledge of ongoing operations and professional judgement when reviewing

validation files, and therefore sampling standards were not yet developed.

● VITA service owners did not consistently complete a field in the validation files to capture

service owner comments to document the rationale for approving requests to exclude non-

compliant service level agreements from contractor performance. Instead, VITA relied on

discussions with the contractors and stakeholders through other forums and meetings. In

addition, VITA deemed completing the field for each item under review may be too time

consuming under the current process.

Additionally, we identified possible staffing and timing constraints in VITA’s review process of

SLAs through the validation file. Specifically, VITA stated that it has a limited number of reviewers

due to each reviewer having to be intimately familiar with the SLAs. VITA added that under the

current validation file review process, it only has approximately two weeks to review all SLA

metrics to confirm compliance and approve exclusions. On any given month, VITA may review

over 450 of 571 SLAs metrics in two weeks. Further challenging VITAs review timeline is that each

SLA metric could have as many as eighty specific items that must be verified.

As a result, VITA stated that it plans to further automate the monthly review process once it

completes its project to remove SLA redundancies and implements. VITA plans to reassess staffing

and review the timeline to determine what adjustments are needed once these efforts are completed.

Specifically, VITA acknowledged that it may not have enough time each month to complete its

validation file review and is continuing to address this concern in multiple ways:

● Reducing SLAs that must be processed each month through the removal of overlapping

and duplicative items, combining where feasible, and leveraging other oversight tools

where appropriate.

● Enforcing the deadlines in the contract for suppliers to submit information to VITA.

● Modifying future contracts, and working to modify current contracts, to move the

suppliers’ delivery date and correspondingly the MSI Delivery to VITA date.

2023-PA-008

OFFICE OF THE STATE INSPECTOR GENERAL

Criteria: ITIL recommends for organizations to ensure that delivery of services is assessed,

monitored, and managed against these targets:

● Provides the end-to-end visibility of the organization’s services, and ensures the organization

meets the defined service levels through the collection, analysis, storage, and reporting of the

relevant metrics for the identified services.

● Performs service reviews to ensure that the current set of services continues to meet the

needs of the organization and its customers.

● Captures and reports on service issues, including performance against defined service levels.

Without effective internal controls over SLA monitoring processes, VITA may not be able to ensure

that contractor performance is adequately reviewed and verified. For example, inconsistent sampling

procedures and documentation of completed reviews of requested exclusions increases the risk that

oversight of SLA compliance does not meet VITA mission requirements.

Recommendations:

1. Once Project Evolution and the implementation of the latest version of SLA 2.3.4 – Security

& Vulnerability Patching are completed, evaluate the processes used to review the Monthly

Validation Files to identify automation where possible.

2. Evaluate the current contracts for the Service Tower Providers and determine if the contract

requirements and timelines associated with the Monthly Validation File should be modified

to provide VITA with additional review time.

3. Based on the evaluations in Recommendations 1 and 2, VITA should review its current

resources for monitoring and managing contract requirements and enhance its resources

and/or staffing as needed.

4. Determine if sampling of SLA items for review in the Monthly Validation File is appropriate

and document the methodology that should be used if sampling is allowed.

5. Update the Monthly Validation File review process to formalize and standardize service

owner comments for unmet SLAs metrics that are submitted for exclusion.

VITA’s Response:

1. VITA will continue to replace the current, mostly manual processes with increased

automation of the monthly processing, where available tools and technology use are feasible.

Efforts undertaken will be documented as improvement efforts and tracked as such.

2a. As part of Project Evolution, VITA is requesting a change to the delivery date of the data.

This is the first step that once implemented and coupled with automation will be reassessed

to determine if additional steps are required.

2b. Re-evaluate and document additional recommended changes.

3. VITA will gather and assess evaluation timeframes and will notate and address resource

needs.

4. VITA will document expectations for the development of sampling methodology to be used

by each reviewer.

2023-PA-008

OFFICE OF THE STATE INSPECTOR GENERAL

5. VITA will update the appropriate documentation and work instructions around Service

Owner comments related to SLA exceptions. VITA has already worked with MSI and

implemented a specified field for recording the reviewer(s) of the file.

Estimated Completion Date: August 2024

2023-PA-008

OFFICE OF THE STATE INSPECTOR GENERAL

Appendix – VITA Corrective Action Plan

VITA Corrective Action Plan

Finding

No.

Recommendation

Corrective Action

Deliverable

Estimated

Completion

Date

Responsible

Position

1. Review those

Tower Service

Levels that do not

have a monetary

impact and remove

those that are not

deemed to add

value to COV

agencies. In

addition, VITA

should minimize

Tower service level

requirements that

do not include

monetary impact

1. VITA is

removing over half

of all SLAs and

removing the

category of SLAs

that do not have

monetary impact

with Project

Evolution. This

modification also

is moving the

program to a single

target for all SLAs,

removing the need

to measure each

SLA 2 times, once

for Target and

once for

Minimum.

1a. Delivery of

new SLM related

exhibits to all

Suppliers

August 2023

Manager,

Performance

& Data

Analytics

1b. Update of

measurement

tools to the

single

measurement for

all SLAs

August 2023

Manager,

Performance

& Data

Analytics

2. Develop a

formalized process

to evaluate SLAs

for duplicative and

inefficient metrics

at least annually

and seek input from

COV agencies in

this process.

2a. VITA will

develop a review

cycle that will

include input from

the agencies

through the

Relationship

Management

Committee

(RMC). The

effectiveness of

this process will be

dependent upon

the completion of

the implementation

of Project

Evolution.

2. Deliver a

review Process

leveraging the

appropriate tools

& forums

June 2024

Manager,

Performance

& Data

Analytics

2023-PA-008

OFFICE OF THE STATE INSPECTOR GENERAL

Finding

No.

Recommendation

Corrective Action

Deliverable

Estimated

Completion

Date

Responsible

Position

2b. Any

recommendations

for related changes

based on this

review will be

reviewed with the

Service Owners

and presented to

the SLM Forum in

one of the Monthly

SLM Forum

meetings for

decision.

1. Develop a

process to formally

compare customer

feedback to SLAs

on a periodic basis

to identify trends

and adjustments

that should be made

to SLAs.

1a. VITA has

developed a

handoff process

between the

Customer

Satisfaction

Review Team and

the SLM Team.

1a. Handoff

process from

Survey Team to

SLM Team

Complete

Manager,

Performance

& Data

Analytics

1b. Research and

determine the

methodology.

Then develop a

work instruction

for the SLM Team

that will be used to

evaluate trends and

adjustments based

on the feedback

received.

1b. Work

Instruction

updates

March 2024

Manager,

Performance

& Data

Analytics

2. Develop a

mechanism to

formally share the

expected metric

requirements with

agencies that use

Service Tower

support contractors.

2. VITA will

develop a

mechanism that

makes SLA

definitions and

metrics more

easily available for

agency review.

This is dependent

on completion of

Project Evolution

and will require

resources outside

the SLM team to

implement

2. Portal or other

accessible

mechanism for

seeing/reviewing

definitions and

measurements in

the SLM

portfolio

May 2024

Manager,

Performance

& Data

Analytics.

2023-PA-008

OFFICE OF THE STATE INSPECTOR GENERAL

Finding

No.

Recommendation

Corrective Action

Deliverable

Estimated

Completion

Date

Responsible

Position

1. Once Project

Evolution and the

implementation of

the latest version of

SLA 2.3.4 –

Security &

Vulnerability

Patching are

completed, evaluate

the processes used

to review the

Monthly Validation

Files to identify

automation where

possible.

1. VITA will

continue to replace

the current, mostly

manual processes

with increased

automation of the

monthly

processing, where

available tools and

technology use are

feasible. Efforts

undertaken will be

documented as

improvement

efforts and tracked

as such.

1a. Complete

automation of

Validation File

review decisions

December 2023

Manager,

Performance

& Data

Analytics.

1b. Create

documented plan

for next phases

of automation

Project

Evolution go-

live +6 and +12

months

Manager,

Performance

& Data

Analytics.

2. Evaluate the

current contracts

for the Service

Tower Providers

and determine if the

contract

requirements and

timelines associated

with the Monthly

Validation File

should be modified

to provide VITA

with additional

review time.

2a. As part of

Project Evolution,

VITA is requesting

a change to the

delivery date of the

data. This is the

first step that once

implemented and

coupled with

automation will be

reassessed to

determine if

additional steps are

required.

2a. New delivery

date

implemented

August 2023

(currently

pending

supplier

signatures)

Manager,

Performance

& Data

Analytics.

2b. Re-evaluate

and document

additional

recommended

changes

2b. Follow-up

assessment of

impact of

automaton and

identify &

document

additional needs

August 2024

(evaluated

quarterly over

6-12 mos. after

implementation

date of 2a)

Manager,

Performance

& Data

Analytics.

2023-PA-008

OFFICE OF THE STATE INSPECTOR GENERAL

Finding

No.

Recommendation

Corrective Action

Deliverable

Estimated

Completion

Date

Responsible

Position

3. Based on the

evaluations in

Recommendations

1 and 2, VITA

should review its

current resources

for monitoring and

managing contract

requirements and

enhance its

resources and/or

staffing as needed.

3. VITA will

gather and assess

evaluation

timeframes and

will notate and

address resource

needs

3. Provide a

current resource

assessment

following Project

Evolution and

completion of

Validation File

intake

automation

May 2024

Chief of Core

Infrastructure

Services

4. Determine if

sampling of SLA

items for review in

the Monthly

Validation File is

appropriate and

document the

methodology that

should be used if

sampling is

allowed.

4. VITA will

document

expectations for

the development of

sampling

methodology to be

used by each

reviewer

4. Documented

expectations for

development of

sampling

methodology.

June 2024

Manager,

Performance

& Data

Analytics.

5. Update the

Monthly Validation

File review to

formalize and

standardize service

owner comments

for unmet SLAs

metrics that are

submitted for

exclusion.

5. VITA will

update the

appropriate

documentation and

work instructions

around Service

Owner comments

related to SLA

exceptions. VITA

has already

worked with MSI

and implemented a

specified field for

recording the

reviewer(s) of the

file.

5. Documented

work instructions

for Service

Owner

Comments

December 2023

Manager,

Performance

& Data

Analytics.